A method for establishing a fast image denoising model for robot welding vision

Abstract

A method for establishing an image fast denoising model for robot welding vision belongs to the field of imaging technology. It solves the problem of fast reduction of spatter noise and imaging in the process of robot weld tracking. The present invention first obtains multiple frames of calibration images with different exposure times; calculates the irradiance expression value of each pixel point of all calibration images; calculates the weighted average of the irradiance expression values ​​of the corresponding pixel points of all images as the final irradiance expression value of the corresponding pixel point; compresses the irradiance expression value into a target grayscale value; arbitrarily selects two frames of calibration images with different exposure times, constructs a lookup table with a mapping relationship, and uses graphics operation methods to eliminate data outliers in the table and burrs on the edges of the region; sets the distribution area of ​​the data set as the ridge area, and corrects the ridge area data by the pixel neighborhood grayscale averaging method and the convolution averaging method; and completes the model establishment. The present invention is suitable for fast image denoising.

Description

Technical Field

[0001] The invention belongs to the technical field of imaging. Background Art

[0002] Robotic welding plays an important role in the manufacturing industry, and the imaging system is crucial in weld seam recognition and tracking. In many robot weld seam tracking processes, welding spatter noise is inevitable, which causes overexposure of the image to a certain extent. At the same time, spatter noise may interfere with visual information such as weld seams and guide lasers.

[0003] The invention patent with application publication number CN111299763A proposes a noise-resistant laser vision weld automatic tracking method and system, which solves the problem of strong noise interference in the welding process by adversarial neural network. Another example is the invention patent with application publication number CN111028159A, which corrects noisy images by neural network. However, the application of such methods in welding requires the host computer to have strong computing power and perform complex calculations on the images, which takes a long time. The patent with application publication number CN110072051A obtains multiple frames of images and synthesizes them, and the resulting images have good visual effects in both bright and dark areas. However, the application of such methods also requires a lot of calculations, and the methods focus on static targets, while welding spatter is a moving target, which makes the information of multiple frames of images different and is not applicable.

[0004] LinLog technology (linear logarithmic imaging technology) proposed by PhotonFocus suppresses the photoelectric conversion process at the image sensor level, so that pixels that receive a small number of photons are normally exposed linearly, while pixels that receive a large number of photons approach grayscale saturation at a relatively slow speed, in order to achieve the purpose of high dynamic range imaging. However, the resulting image may have a strong light "blurring" effect, making part of the picture less clear. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of rapid reduction of spatter noise and imaging during robot weld tracking, and provide a method for establishing an image rapid denoising model for robot welding vision.

[0006] The method for establishing a fast image denoising model for robot welding vision described in the present invention specifically comprises:

[0007] Step 1: Obtain multiple calibration images with different exposure times;

[0008] Step 2: Calculate the irradiance expression value of each pixel in all calibration images;

[0009] Step 3: Calculate the weighted average of the irradiance expression values ​​of the corresponding pixels of all images as the final irradiance expression value of the corresponding pixels;

[0010] Step 4: compress the final irradiance expression value into a target grayscale value using a tone mapping function;

[0011] Step 5: arbitrarily select two calibration images with different exposure times, and use the target grayscale value of each corresponding pixel point in the two frames of images to construct a lookup table with a mapping relationship, where the horizontal and vertical coordinates of the lookup table are the pixel grayscale values ​​of images with different exposure times;

[0012] Step 6: Use graphics operation methods to eliminate data outliers and burrs on the edges of regions in the table;

[0013] Step 7: Set the distribution area in the data set as the ridge area, and correct the ridge area data through the pixel neighborhood grayscale averaging method and the convolution averaging method; complete the model establishment.

[0014] Furthermore, in the present invention, the multiple frames of calibration images with different exposure times in step one include at least one frame of image with the shortest exposure time and one frame of image with the longest exposure time, and the exposure times are different, and the industrial camera and the acquisition scene are kept relatively still during the acquisition process.

[0015] Furthermore, in the present invention, the specific method for calculating the irradiance expression value of each pixel point in all calibration images in step 2 is:

[0016] Step 21: Establish a relationship function based on the inherent relationship between pixel grayscale, irradiance and exposure time;

[0017] Step 22: Set the loss function of the relationship function and use the least squares method to obtain the irradiance expression value function;

[0018] Step 2: Substitute the grayscale and exposure time of each pixel of each calibration image into the irradiance expression value function to obtain the irradiance expression value of each pixel of the calibration image.

[0019] Furthermore, in the present invention, the loss function of the relationship function g is set in step 22, and the specific method of obtaining the irradiance expression value function by using the least squares method is:

[0020] Set the loss function of the relationship function:

[0021]

[0022] Among them, g(Z ij )=ln E i +lnΔt j g(Z ij) is the relationship function, g(Z ij )=ln E i +lnΔt j , Δt j is the exposure time of the j-th frame calibration image, E i is the irradiance expression value corresponding to the position of the i-th pixel, Z ij is the grayscale of the i-th pixel of the j-th frame calibration image, w(Z ij ) is the pixel grayscale equal to Z ij The corresponding weight when λ is the function O map The equilibrium parameter between the first and second terms in max is the maximum grayscale value of the pixel, Z min is the minimum grayscale value of the pixel, N is the total number of pixels in a frame, P is the total number of frames of the calibration image, g″(Z ij ) is the smoothing control parameter of the function g curve, at the pixel gray value Z ij The expression at is:

[0023] g″(Z ij )=g(Z ij -1)-2g(Z ij )+g(Z ij +1)

[0024] When the pixel grayscale is close to the extreme value Z max , Z min When , the weight adopts a Sigmoid-like function:

[0025]

[0026] Among them, γ trs Proposed thresholds for need;

[0027] Using the least squares method, solve the relationship function g(Z ij ), and obtain the irradiance expression function:

[0028] E i =exp(g(Z ij )-lnΔt j ).

[0029] Furthermore, the specific method for calculating the weighted average of the irradiance expression values ​​of all corresponding pixels in the image in step 3 is:

[0030] Arrange all calibration images in ascending order according to exposure time from short to long to construct a three-dimensional matrix, wherein the first layer of the three-dimensional matrix is ​​the shortest exposure time frame, and the last layer is the longest exposure time frame;

[0031] Search all saturated pixels from the 1st layer to the nth layer in sequence, record the position of the pixel, and remove the pixel points corresponding to the position of the saturated pixel in the m layers after the layer with the saturated pixel; obtain valid pixel points; where m is an integer greater than 1 and less than n;

[0032] Calculate the irradiance expression value of the effective pixel point at each pixel position in multiple planes in the first and second dimensions of the three-dimensional matrix, and calculate the weighted average of the irradiance expression values, wherein the weight parameter used in the weighted average calculation is the grayscale weight factor.

[0033] Furthermore, the tone mapping function used in step 4 is composed of a nonlinear function, having two dimensions of input and output, and the slope of the function curve formed near the minimum or maximum value of the input dimension is smaller than the slope of the function curve formed in the range near the non-minimum or maximum value.

[0034] Furthermore, in step 5, two calibration images with different exposure times are randomly selected, and the specific method of constructing a lookup table with a mapping relationship using the target grayscale value of each corresponding pixel in the two frames of images is as follows:

[0035] Create an empty two-dimensional matrix, where the size of each dimension of the matrix is ​​equal to the grayscale level of the calibration image;

[0036] The two grayscale values ​​of the same pixel position in the two frames of calibration images with different exposure times are matched to the horizontal and vertical coordinates of the two-dimensional matrix, and the target grayscale of the pixel point is filled into the corresponding position of the two-dimensional matrix.

[0037] Furthermore, step eight is included, and the step eight includes determining whether there are vacancies in the lookup table with a mapping relationship. If there are vacancies, if the vacancies are at the lower left of the ridge area, they are filled with the data of the edge of the ridge area above the corresponding vacancy; if the vacancy is at the upper right of the ridge area, it is filled with the data of the edge of the ridge area to the left of the vacancy.

[0038] The present invention obtains multiple frames of calibration images, screens effective pixels, and restores the irradiance expression value, thereby ensuring the accuracy of the restoration result. By screening effective unsaturated pixels for the second time, pixels that fail when reflecting the irradiance expression value are eliminated, thereby ensuring the resolution of the tone mapping output. By constructing a lookup table, a mapping relationship between input and output is formed. By correcting the lookup table and calibrating its vacant positions, a strategically responsive high-dynamic camera imaging model is formed, so that the calibrated model can effectively respond to any input combination. The model obtained by the calibration method can achieve denoising of spatter noise in robot weld tracking applications, so that the image clarity is maintained at a high level. Since the overall model is based on the lookup table, there is no need to perform time-consuming and complex operations such as image processing or neural networks in the application, and imaging and denoising can be completed quickly at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 Establish a schematic diagram for the model of the present invention;

[0041] Figure 3 An example diagram of the application stage of the method for establishing a fast image denoising model for robot welding vision provided by an embodiment of the present invention;

[0042] Figure 4 This is the effect image of image processing of robot welding vision using LinLog technology. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0045] Specific implementation method 1: The following is combined Figure 1 This embodiment describes the method for establishing a fast image denoising model for robot welding vision, which specifically includes:

[0046] Step 1: Obtain multiple calibration images with different exposure times;

[0047] Step 2: Calculate the irradiance expression value of each pixel in all calibration images;

[0048] Step 3: Calculate the weighted average of the irradiance expression values ​​of the corresponding pixels of all images as the final irradiance expression value of the corresponding pixels;

[0049] Step 4: compress the final irradiance expression value into a target grayscale value using a tone mapping function;

[0050] Step 5: arbitrarily select two calibration images with different exposure times, and use the target grayscale value of each corresponding pixel point in the two frames of images to construct a lookup table with a mapping relationship, where the horizontal and vertical coordinates of the lookup table are the pixel grayscale values ​​of images with different exposure times;

[0051] Step 6: Use graphics operation methods to eliminate data outliers and burrs on the edges of regions in the table;

[0052] Step 7: Set the distribution area in the data set as the ridge area, and correct the ridge area data through the pixel neighborhood grayscale averaging method and the convolution averaging method; complete the model establishment.

[0053] In this implementation, the data center distribution area in step seven is an area composed of all non-empty positions whose neighborhoods can be interconnected. The neighborhood is the 8 coordinate positions adjacent to the coordinate position, and the interconnection means that there is no empty space in the middle.

[0054] Furthermore, in this embodiment, the multiple frames of calibration images with different exposure times in step one include at least one frame of image with the shortest exposure time and one frame of image with the longest exposure time, and the exposure times are different. During the acquisition process, the industrial camera and the acquisition scene are kept relatively still.

[0055] In this implementation, the shortest exposure time is not shorter than the unit exposure time of the camera, and the longest exposure time should not cause all pixels of the frame image to be overexposed.

[0056] Furthermore, in this embodiment, the specific method for calculating the irradiance expression value of each pixel point in all calibration images in step 2 is:

[0057] Step 21: Establish a relationship function g based on the inherent relationship between pixel grayscale, irradiance and exposure time;

[0058] Step 22: Set the loss function of the relationship function g, and use the least squares method to obtain the irradiance expression value function;

[0059] Step 2: Substitute the grayscale and exposure time of each pixel of each calibration image into the irradiance expression value function to obtain the irradiance expression value of each pixel of the calibration image.

[0060] Furthermore, in step 22, the loss function of the relationship function is set, and the specific method of obtaining the irradiance expression value function using the least squares method is:

[0061] Set the loss function of the relationship function:

[0062]

[0063] Among them, g(Z ij )=ln E i +lnΔt j g(Z ij ) is the relationship function, g(Z ij )=ln E i +lnΔt j , Δt j is the exposure time of the j-th frame calibration image, E iis the irradiance expression value corresponding to the position of the i-th pixel, Z ij is the grayscale of the i-th pixel of the j-th frame calibration image, w(Z ij ) is the pixel grayscale equal to Z ij The corresponding weight when λ is the function O map The equilibrium parameter between the first and second terms in max is the maximum grayscale value of the pixel, Z min is the minimum grayscale value of the pixel, N is the total number of pixels in a frame, P is the total number of frames of the calibration image, g″(Z ij ) is the smoothing control parameter of the function g curve, at the pixel gray value Z ij The expression at is:

[0064] g″(Z ij )=g(Z ij -1)-2g(Z ij )+g(Z ij +1)

[0065] When the pixel grayscale is close to the extreme value Z max , Z min When , the weight adopts a Sigmoid-like function:

[0066]

[0067] Among them, γ trs is the threshold value to be proposed. The larger the value, the stronger the suppression effect. Here, it is 70;

[0068] Using the least squares method, solve the relationship function g(Z ij ), and obtain the irradiance expression function:

[0069] E i =exp(g(Z ij )-lnΔt j ).

[0070] Furthermore, the specific method for calculating the weighted average of the irradiance expression values ​​of all corresponding pixels in the image in step 3 is:

[0071] Arrange all calibration images in ascending order according to exposure time from short to long to construct a three-dimensional matrix, wherein the first layer of the three-dimensional matrix is ​​the shortest exposure time frame, and the last layer is the longest exposure time frame;

[0072] Search all saturated pixels from the 1st layer to the nth layer in sequence, record the position of the pixel, and remove the pixel points corresponding to the position of the saturated pixel in the m layers after the layer with the saturated pixel; obtain valid pixel points; where m is an integer greater than 1 and less than n;

[0073] Calculate the irradiance expression value of the effective pixel point at each pixel position in multiple planes in the first and second dimensions of the three-dimensional matrix, and calculate the weighted average of the irradiance expression values, wherein the weight parameter used in the weighted average calculation is the grayscale weight factor.

[0074] Furthermore, the tone mapping function used in step 4 is composed of a nonlinear function, having two dimensions of input and output, and the slope of the function curve formed near the minimum or maximum value of the input dimension is smaller than the slope of the function curve formed in the range near the non-minimum or maximum value.

[0075] Furthermore, in step 5, two calibration images with different exposure times are randomly selected, and the specific method of constructing a lookup table with a mapping relationship using the target grayscale value of each corresponding pixel in the two frames of images is as follows:

[0076] Create an empty two-dimensional matrix, where the size of each dimension of the matrix is ​​equal to the grayscale level of the calibration image;

[0077] The two grayscale values ​​of the same pixel position in the two frames of calibration images with different exposure times are matched to the horizontal and vertical coordinates of the two-dimensional matrix, and the target grayscale of the pixel point is filled into the corresponding position of the two-dimensional matrix.

[0078] Furthermore, step eight is included, and the step eight includes determining whether there are vacancies in the lookup table with a mapping relationship. If there are vacancies, if the vacancies are at the lower left of the ridge area, they are filled with the data of the edge of the ridge area above the corresponding vacancy; if the vacancy is at the upper right of the ridge area, it is filled with the data of the edge of the ridge area to the left of the vacancy.

[0079] In this embodiment, the lookup table is also corrected and its empty positions are filled, including: performing graphics operations on the area formed by the non-empty positions where data exists in the lookup table, eliminating outliers and burrs on the edges of the area, and defining the remaining area as the ridge area; according to the prior condition, smoothing the pixel grayscale of the ridge area so that the grayscale change of the ridge area conforms to the exposure law.

[0080] In one embodiment of the present invention, the step of correcting the lookup table and filling the missing positions thereof further comprises:

[0081] Find the missing areas in the lookup table except the ridge area;

[0082] When the short exposure coordinate value of a vacant position is greater than the maximum short exposure coordinate value on the ridge line corresponding to its long exposure coordinate value, the grayscale of the combination position of the maximum short exposure coordinate value and the long exposure coordinate value of the vacant position is assigned to the vacant position;

[0083] When the long exposure coordinate value of a certain vacant position is greater than the maximum long exposure coordinate value on the ridge line corresponding to its short exposure coordinate value, the grayscale of the combined position of the short exposure coordinate value of the vacant position and the maximum long exposure coordinate value is assigned to the vacant position;

[0084] All coordinate positions where the short exposure coordinates and the long exposure coordinates are the minimum values ​​are set to the expected minimum grayscale level, and the remaining vacant positions are smoothly interpolated to complete the calibration of the imaging model.

[0085] Specific embodiment: Combination Figures 2 to 4 Describe this embodiment:

[0086] High dynamic camera imaging model calibration for fast noise reduction in robotic weld tracking, applied to imaging processes with splash noise. Figure 2 , Figure 2 A schematic diagram is established for the model described in the present invention, and is used in a high-dynamic camera imaging model for robot weld tracking and rapid noise reduction, including:

[0087] Step 101, obtaining multiple frames of calibration images.

[0088] In this embodiment, the acquisition of multiple calibration images includes two or more frames of images with different exposure times, including at least one frame of the shortest exposure time image and one frame of the longest exposure time image, and the industrial camera and the acquisition scene are kept relatively still during the acquisition process. For industrial camera parameters, except for the difference in exposure time, the settings of other parameters are consistent with the parameters in actual application.

[0089] Step 102, synthesize the calibration image to obtain the irradiance expression value.

[0090] Step 103: extract any two calibration image frames, compress the irradiance expression value, and construct a lookup table with a mapping relationship between the compression result and the two calibration image frames.

[0091] Step 104, correcting the lookup table and filling in the missing positions to form an imaging model.

[0092] In order to make the calibration more accurate and suitable for the actual application of strong light in weld tracking, further requirements can be made on the acquisition environment, the data of the collected calibration images can be reasonably used, and the reflection of the real light intensity of the target object can be calculated through multiple frames of calibration images. Figure 2 A schematic flow chart of a second high-dynamic camera imaging model calibration method for robot weld tracking and rapid noise reduction provided by an embodiment of the present invention includes:

[0093] Step 201 , selecting a capture scene so that the camera field of view includes strong light and weak light close to the weld tracking scene.

[0094] In this embodiment, the acquisition process utilizes the industrial camera's own functions or an external device to trigger the industrial camera's electronic shutter and control the exposure time, and quickly completes the acquisition of multiple frames of calibration images. As the acquisition process begins and ends, the exposure time of the calibration images does not need to be arranged in ascending order. However, considering the convenience of subsequent data sorting, it is recommended to collect in ascending order according to the exposure time.

[0095] The acquisition scene includes a strong light area and a weak light area. The extreme value of the light radiation illuminance of the target object within the field of view of the industrial camera conforms to the extreme value of the light radiation illuminance of the final application scene. The target object corresponding to the strong light area is in a state of stable irradiance. The acquisition scene is arranged using natural light sources or artificial light sources, even if part of the field of view of the industrial camera contains sunlight or a strong light flashlight aimed at the industrial camera.

[0096] The industrial camera should include a program interface that can be developed for secondary development or an IO terminal that can be controlled by an external device to start and end exposure. It should be noted that the default function of the camera usually does not have the function of variable exposure in step 101. An effective method is to connect the camera IO through the FPGA development board, design the exposure time and duration (high-level signal) and the image data readout time and duration (low-level signal) through the VerilogHDL language, burn the bitstream file into the FPGA, and then set the camera to the pulse width trigger exposure mode to prepare for acquisition. One point to note is that the image readout time varies depending on the camera model. The low-level signal should be kept short enough (to ensure that the overall acquisition process is completed quickly) but not too short, otherwise the next frame will be read out when the previous frame is not fully read out, which may cause frame loss. The duration design of the low-level signal should be obtained through multiple experiments. Another point to note is that the industrial camera IO trigger voltage may be 12 volts or other values ​​greater than 3.3 volts. The FPGA or other development board IO generally provides a 3.3 volt output. Here, voltage conversion can be performed through a high-frequency optical coupler.

[0097] Step 202, setting different exposure times and collecting calibration images.

[0098] In this embodiment, the exposure time is different. If the number of calibration images exceeds two frames, after all images are arranged in ascending order according to the exposure time, the exposure time ratio of all two adjacent frames should be the same. For example, the first frame is exposed for 10 microseconds as the shortest exposure time image, and the next frame is exposed at a ratio of twice the exposure time of the previous frame, the second frame is 20 microseconds, the third frame is 40 microseconds, and until the twelfth frame is exposed for 20.48 milliseconds, a set of calibration images is acquired. Of course, in order to make the pixel grayscale in the acquired calibration image not accidental, two sets of calibration images can be acquired continuously, and the error can be reduced by the mean method.

[0099] Step 203 , filter out valid pixels that are not saturated in the calibration image with a shorter exposure time.

[0100] In this embodiment, in order to prevent the grayscale saturation of pixels in the shortest exposure time frame, which will reduce the grayscale reference significance of other frames at the pixel position, it is necessary to effectively screen the pixels of the calibration image. For the exposure law, the irradiance E remains unchanged, the larger the exposure time Δt, the larger the exposure X, and the larger the grayscale of the obtained image. When the grayscale saturation appears in the frame with a shorter exposure time, the frame with an exposure longer than the time will usually be in a saturated state. Except for the frame that happens to be saturated and the frame with an exposure time shorter than the frame, the other frames cannot reflect the true state of X. One thing to note is that due to the extremely strong welding arc light intensity, even the frame with the shortest exposure time may still have saturated grayscale. The same situation applies to the acquisition scene used to simulate the welding arc light intensity. If the irradiance E is calculated using twelve frames of the same pixel, most frames may mislead the results.

[0101] Here, all calibration images are arranged in ascending order from short to long exposure time, and assigned to a new three-dimensional matrix. The third dimension of the matrix is ​​the time dimension. The first layer is the shortest exposure time frame, and the last layer is the longest exposure time frame. From the first layer to the nth layer, search for saturated pixels. If there are saturated pixels, record the position of the pixel in the nth layer, and delete the pixels at this position in all layers after n plus 2 layers. The remaining pixels in all frames are valid pixels. When the total number of layers is twelve, n is set to a maximum of 3.

[0102] Step 204, calculating the irradiance expression value through the effective pixels in the multiple frames of calibration images.

[0103] In this embodiment, according to the inherent relationship between pixel grayscale, irradiance and exposure time, a function g is established and written as: g(Z ij )=ln E i +lnΔt j , where Δt j is the exposure time of the j-th frame calibration image, E iis the irradiance expression value corresponding to the position of the i-th pixel, Z ij The grayscale of the i-th pixel of the j-th frame calibration image is calculated based on the performance characteristics of the common imaging device with a binary 8-bit depth. ij The value range is from the minimum value Z min To the maximum value Z max Set from 0 to 255, g is the function ln f -1 The function f -1 is the inverse function of function f, which is the exposure from X to Z ij The exposure X is E i With Δt j The product of .

[0104] In the g function, Z ij , Δt j is a known quantity, and the unknown quantities include E i and g(Z ij ). According to the cognitive law, the stronger the exposure X is, the greater the grayscale presented by the camera is. It is stipulated that f is a monotonically increasing function, and the inverse function f is stipulated at the same time. -1 Monotonically increasing, and then stipulate ln f -1 That is, the g function is monotonically increasing. By setting the loss function of the g function and minimizing the loss function, we can find g(Z ij ).

[0105] Set the loss function expression to: Among them, w(Z ij ) is the pixel grayscale equal to Z ij The corresponding weight when the pixel grayscale is close to the extreme value Z max , Z min When , it is considered that the mapping result is better than Z near the non-extreme value ij More unstable, during the calculation process, such data is suppressed, giving smaller weights to the grayscale near the extreme values ​​and larger weights to the grayscale near the non-extreme values.

[0106] Where g″(z) is the smoothing control parameter of the function g curve, and its expression at z is g″(z)=g(z-1)-2g(z)+g(z+1), and λ is the function O map The balance parameter between the first and second terms in , increasing λ makes the smoothing control parameter play a greater role in the loss function minimization process. The weight adopts a Sigmoid function Among them, γ trsis the threshold value. The larger its value, the stronger the suppression effect. Here, it is 70. The least square method is used for the loss function to obtain the function g. It should be noted that since there is no reference value for the actual irradiance, in the process of using the least square method, Z is calibrated. ij The midpoint value of The result is called the irradiance expression value, E i =exp(g(Z ij )-lnΔt j ).

[0107] For each pixel position in each frame of the calibration image corresponding to the acquisition scene, the irradiance expression value E is calculated using the obtained g function curve and the known exposure time prior conditions and pixel grayscale. i The irradiance expression values ​​of all the filtered effective pixels are combined by weighted average to obtain the final irradiance expression value, where the weight is still the w(Z ij ).

[0108] Step 301, select any two calibration image frames and retain unsaturated pixels.

[0109] In this embodiment, two calibration images are selected from all calibration images, which are recorded as short exposure images and long exposure images. The purpose of selecting two frames is to calibrate the output values ​​when the two frames are used as input values. In the application scenario, it is planned to alternately perform two exposure durations. For example, 40 microseconds and 320 microseconds frames are alternately exposed. The first frame with an exposure duration of 40 microseconds is used as the short exposure duration frame of the first pair of inputs, and the 320 microsecond frame is used as the long exposure duration frame of the first pair of inputs. When the second exposure duration is 40 microseconds, the frame will form the second pair of inputs together with the 320 microsecond frame in the first pair of inputs.

[0110] The two durations of each pair of inputs should be designed in combination with practical applications. A feasible method is to pair all twelve calibration images in pairs. There are 11 combinations paired with the first frame, 10 other combinations paired with the second frame, and so on, for a total of 66 combinations. The selection of any two calibration images is among the 66 combinations. It should be noted that the following operations can be performed on all combinations, and then the results of each combination are stored in the memory unit, but usually some combinations are not selected because the corresponding inputs do not show obvious differences. In practical applications, this combination is not selected as the two exposure durations of a pair of inputs. For example, the results of 20.48 milliseconds and 10.24 milliseconds exposure durations may not be much different, and the grayscale of most areas of the image is saturated and has no reference value.

[0111] The irradiance expression values ​​of the short exposure image, long exposure image, and acquisition scene are reduced in dimension to form an array of q rows and one column. The pixel positions correspond and are merged into an array of q rows and three columns. The first column is searched to see if there is a pixel with saturated grayscale. If so, the irradiance expression value corresponding to the pixel is deleted. The saturated area in a frame is usually multiple pixels. All saturated grayscale positions in the short exposure frame may be just saturated or oversaturated (but because the maximum grayscale value is reached, only the maximum grayscale can be displayed), and the corresponding irradiance expression values ​​are not the same. Therefore, the irradiance expression value of this area does not have a theoretical single mapping relationship in the two selected calibration images. Here, the short and long exposure input and irradiance expression values ​​of this type are invalidated and replaced by the average of all irradiance expression values ​​corresponding to the pixel positions in the short exposure image that are one grayscale less than the saturated grayscale. At this time, the retained pixels have one less grayscale level than before, but it is conducive to mapping all unsaturated pixels to the ideal output domain, ensuring the resolution of the tone mapping output.

[0112] Step 302, designing a tone mapping function with reference to two calibration images.

[0113] The adjusted irradiance expression value is compressed through a tone mapping function and converted into a grayscale value that can be displayed by a common imaging device. The tone mapping function is composed of a power function and a logarithmic function, respectively: y = a*x 2 , 0<x≤x1; y=a1*ln(x)+b1, x1<x≤x2; y=a2*ln(x)+b2, x2<x≤x max Where x is the irradiance expression value. To ensure that the strong light area and weak light area of ​​the image are compressed to a smaller range, and the medium light intensity area is compressed to a larger visible range, here, x1 is set to 2 times the minimum value of the third column of the array, and x2 is set to 3 times the average value of the first column of the array, and the value of the third column of the array is corresponding to the integer. max is the maximum value of the third column of the array, y|(x=x1) is set to 10, y|(x=x2) is set to 200, where a, a1 and a2 are the coefficients of the above-mentioned power function and logarithmic function respectively, and b1 and b2 are constants.

[0114] Step 303: compress the irradiance expression value corresponding to the retained pixel to the target grayscale.

[0115] All irradiance expression values ​​are brought into the tone mapping function to perform numerical compression.

[0116] Step 304, using the selected image and the compression result corresponding to the pixel position thereof as input and output respectively, to establish a lookup table.

[0117] In this embodiment, an empty two-dimensional square matrix is ​​established, and the size of each dimension of the matrix is ​​equal to the grayscale level of the exposure image. When the grayscale level is 256, the matrix size is 256*256. The first and second columns of the m-row and three-column array are made unique, and all rows with the same first column elements and the same second column elements are extracted and recorded as and in, Input grayscale for short exposure time, Input the grayscale for the long exposure time, calculate the average value of the third column elements corresponding to them, the value range should be 0 to 255, fill the average value into the empty two-dimensional square matrix, and the horizontal and vertical coordinates of the filled position are and Form a lookup table, where when the matrix size is 256*256, and The value ranges from 1 to 256;

[0118] The lookup table is denoted as in, represents the search operation from input to output, f (2) Indicates the output grayscale.

[0119] The process of processing non-empty areas of a lookup table.

[0120] Step 401: Graphics operations are performed on non-empty areas of the lookup table.

[0121] In this embodiment, a graphic operation is performed on the area formed by the non-empty positions where data exists in the lookup table, and the obvious outliers and burrs on the edge of the area are eliminated to make the edge of the area smooth, and the area where most data is distributed is defined as the ridge area. Among them, since there are inevitably accidental items in the input combination of the lookup table, that is, and Due to factors such as camera dark noise and slight changes in the acquisition scene, the distribution of some combined coordinate points leaves the ridge area. Operations such as corrosion and expansion are used to eliminate such data.

[0122] Step 402: modify the non-empty area of ​​the lookup table using the priori condition.

[0123] In this embodiment, the non-empty area is corrected to make the ridge line more reasonable in value. Taking 8-bit deep pixels as an example, arrive The output value of the lookup table shows an increasing trend. According to the prior condition, the grayscale of those pixel positions whose grayscale derivatives do not conform to the increasing trend is deleted, and then the positions without data in the ridge area are filled by the method of pixel neighborhood grayscale averaging, and the pixel grayscale of the ridge area is convolutionally averaged to make the grayscale value in the ridge area conform to the increasing trend.

[0124] One thing to note is that in the original lookup table The data should not be reduced in range due to graphics operations, especially the maximum value should not be reduced. The data should not be The data is approximated, that is, the width of the ridge area is The width of other ridge regions should be greater than or equal to 1.

[0125] The larger the short and long exposure grayscale is, the larger the grayscale of the mapped composite image should be. Therefore, in order to ensure that the grayscale of the maximum value position of another coordinate axis on the ridge area corresponding to each position on each coordinate axis of the lookup table increases as the coordinate value of the coordinate axis increases, a row-by-row and column-by-column search method is used to delete the points on the edge of the ridge area that do not meet the prior conditions.

[0126] Step 403 , comprehensively considering the noise characteristics in the weld tracking scenario, strategically fill the empty areas of the lookup table, and complete the calibration of the expected output result.

[0127] In this embodiment, the vacant positions in the lookup table are filled. Considering the characteristics of the spatter in the weld tracking scene that is usually high in brightness and fast in speed, the same spatter generally does not appear in the same position in two consecutive frames of images. Therefore, if the grayscale of a certain position in a certain frame and the grayscale of the same position in another frame appear in the welding image acquisition, and the combined coordinate value does not fall into the ridge area, the pixel input with a larger grayscale is artificially eliminated, and the output value is obtained by the grayscale input of another frame of the image. The lookup table is filled according to this priori condition, and the vacant positions are filled. Perform the following assignments:

[0128] When In the case

[0129] When In the case

[0130] when In the case of ridgeline areas

[0131] in The maximum grayscale value among all possible short exposure time grayscales in the ridge area when the long exposure time input grayscale is known, The maximum grayscale value among all possible long exposure time grayscales in the ridge area when the short exposure time input grayscale is known.

[0132] When the short exposure coordinate value of a vacant position is greater than the maximum short exposure coordinate value on the ridge line corresponding to its long exposure coordinate value, the grayscale of the combined position of the maximum short exposure coordinate value and the long exposure coordinate value of the vacant position is assigned to the vacant position; when the long exposure coordinate value of a vacant position is greater than the maximum long exposure coordinate value on the ridge line corresponding to its short exposure coordinate value, the grayscale of the combined position of the short exposure coordinate value of the vacant position and the maximum long exposure coordinate value is assigned to the vacant position.

[0133] When the coordinates of the input combination are in the ridge region, the output result does not need to be aligned with other values.

[0134] All coordinate positions where the short exposure coordinates and the long exposure coordinates are the minimum values ​​are set to the expected minimum grayscale. and The grayscale of the coordinate position is set to zero, and the remaining vacant positions are smoothly interpolated. After the above processing, the calibration of the high-dynamic camera imaging model with fast noise reduction is completed.

[0135] One thing that needs to be noted is that due to the non-repeatability of real scenes with factors such as electric arcs, spatter, and smoke, there cannot be another identical image for any frame, and the displayed comparison image cannot present exactly the same scene as the compared image. However, all the images displayed reflect the general situation in the experiment and are representative to a certain extent.

[0136] Under real welding conditions, the effect of image processing of robot welding vision using LinLog technology is used as a comparison image, such as Figure 4 As shown in the figure, LinLog technology suppresses the photoelectric conversion process at the image sensor level, so that pixels that receive a small amount of photons are normally exposed linearly, while pixels that receive a large amount of photons approach grayscale saturation at a relatively slow speed, so as to achieve the purpose of high dynamic range imaging. Laplacian gradient function, Brenner gradient function, SMD, etc. are evaluation indicators of image clarity. The comparison effect is shown in the table below:

[0137] Long exposure image Short exposure image Imaging results Compare images Laplacian <![CDATA[1.1789*10 7 ]]> <![CDATA[1.0837*10 7 ]]> <![CDATA[1.0921*10 7 ]]> <![CDATA[7.1274*10 6 ]]> Brenner <![CDATA[8.3963*10 5 ]]> <![CDATA[2.7137*10 5 ]]> <![CDATA[1.158712*10 6 ]]> <![CDATA[4.11514*10 5 ]]> SMD <![CDATA[2.44895*10 5 ]]> <![CDATA[1.90821*10 5 ]]> <![CDATA[2.87741*10 5 ]]> <![CDATA[8.7982*10 4 ]]> SMD2 <![CDATA[2.28262*10 5 ]]> <![CDATA[8.4698*10 4 ]]> <![CDATA[2.6452*10 5 ]]> <![CDATA[8.2786*10 4 ]]> Variance function <![CDATA[2.8543*10 4 ]]> <![CDATA[2.6576*10 5 ]]> <![CDATA[3.0164*10 4 ]]> <![CDATA[1.7937*10 4 ]]> Entropy function <![CDATA[1.34715*10 -1 ]]> <![CDATA[7.289267*10 -2 ]]> <![CDATA[6.964143*10 -2 ]]> <![CDATA[5.864803*10 -2 ]]>

[0138] As can be seen from the above table, the clarity of the imaging results after using the model of the present invention is better than the comparison image, and under the first five evaluation indicators, its clarity is better than all or one of the input exposure images. The model obtained based on the calibration method gives input data in the test and outputs imaging results. The running speed is 95 frames / second on the i5-7300 mobile CPU and 990 frames / second on the GTX1050 mobile GPU. For the working performance of 60 frames / second of ordinary imaging equipment, this imaging speed basically meets the real-time requirements.

[0139] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A method for establishing a rapid image denoising model for robot welding vision, characterized in that: Specifically include: Step 1: Obtain multiple calibration images with different exposure times; Step 2: Calculate the irradiance expression value of each pixel in all calibration images; Step 3: Calculate the weighted average of the irradiance expression values ​​of the corresponding pixels of all images as the final irradiance expression value of the corresponding pixels; Step 4: compress the final irradiance expression value into a target grayscale value using a tone mapping function; Step 5: arbitrarily select two calibration images with different exposure times, and use the target grayscale value of each corresponding pixel point in the two frames of images to construct a lookup table with a mapping relationship, where the horizontal and vertical coordinates of the lookup table are the pixel grayscale values ​​of images with different exposure times; Step 6: Use graphics operation methods to eliminate data outliers and burrs on the edges of regions in the table; Step 7: Set the distribution area of ​​the data set as the ridge area, and correct the ridge area data by using the pixel neighborhood grayscale average method and the convolution average method; complete the model establishment; In step 2, the specific method for calculating the irradiance expression value of each pixel in all calibration images is as follows: Step 21: Establish a relationship function based on the inherent relationship between pixel grayscale, irradiance and exposure time; Step 22: Set the loss function of the relationship function and use the least squares method to obtain the irradiance expression value function; Step 2: Substitute the grayscale and exposure time of each pixel of each calibration image into the irradiance expression value function to obtain the irradiance expression value of each pixel of the calibration image; In step 22, the loss function of the relationship function is set, and the specific method of obtaining the irradiance expression value function using the least squares method is: Set the loss function of the relationship function: Among them, g(Z ij ) is the relationship function, g(Z ij )=lnE i +lnΔt j , Δt j is the exposure time of the j-th frame calibration image, E i is the irradiance expression value corresponding to the position of the i-th pixel, Z ij is the grayscale of the i-th pixel of the j-th frame calibration image, w(Z ij ) is the pixel grayscale equal to Z ij The corresponding weight when λ is the function O map The equilibrium parameter between the first and second terms in max is the maximum grayscale value of the pixel, Z min is the minimum grayscale value of the pixel, N is the total number of pixels in a frame, P is the total number of frames of the calibration image, g”(Z ij ) is the smoothing control parameter of the function g curve, at the pixel gray value Z ij The expression at is: g”(Z ij )=g(Z ij -1)-2g(Z ij )+g(Z ij +1) When the pixel grayscale is close to the extreme value Z max , Z min When , the weight adopts a Sigmoid-like function: Among them, γ trs Proposed thresholds for need; Using the least squares method, solve the relationship function g(Z ij ), and obtain the irradiance expression function: E i =exp(g(Z ij )-lnΔt j )。 2. The method for establishing a rapid image denoising model for robot welding vision according to claim 1, characterized in that: The multiple frames of calibration images with different exposure times in step 1 include at least one frame of image with the shortest exposure time and one frame of image with the longest exposure time, and the exposure times are different. During the acquisition process, the industrial camera and the acquisition scene are kept relatively still.

3. A method for establishing a rapid image denoising model for robot welding vision according to claim 1 or 2, characterized in that: The specific method for calculating the weighted average of the irradiance expression values ​​of all pixels corresponding to the images in step 3 is: Arrange all calibration images in ascending order according to exposure time from short to long to construct a three-dimensional matrix, wherein the first layer of the three-dimensional matrix is ​​the shortest exposure time frame, and the last layer is the longest exposure time frame; Search all saturated pixels from the 1st layer to the nth layer in sequence, and remove the pixels at the positions of the saturated pixels in the m layers after the layer with the saturated pixels; obtain valid pixels; wherein m is an integer greater than 1 and less than n; Calculate the irradiance expression value of the effective pixel point at each pixel position in multiple planes in the first and second dimensions of the three-dimensional matrix, and calculate the weighted average of the irradiance expression values, wherein the weight parameter used in the weighted average calculation is the grayscale weight factor.

4. The method for establishing a rapid image denoising model for robot welding vision according to claim 1, characterized in that: The tone mapping function used in step 4 is composed of a nonlinear function, which contains two dimensions of input and output. The slope of the function curve formed near the minimum or maximum value of the input dimension is smaller than the slope of the function curve formed near the non-minimum or maximum value.

5. The method for establishing a rapid image denoising model for robot welding vision according to claim 1, characterized in that: In step 5, two calibration images with different exposure times are randomly selected, and the specific method of constructing a lookup table with a mapping relationship using the target grayscale value of each corresponding pixel in the two frames of images is as follows: Create an empty two-dimensional matrix, where the size of each dimension of the matrix is ​​equal to the grayscale level of the calibration image; The two grayscale values ​​of the same pixel position in the two frames of calibration images with different exposure times are matched to the horizontal and vertical coordinates of the two-dimensional matrix, and the target grayscale of the pixel point is filled into the corresponding position of the two-dimensional matrix.

6. The method for establishing a rapid image denoising model for robot welding vision according to claim 1, characterized in that: It also includes step eight, which includes determining whether there is a vacancy in the lookup table with a mapping relationship. If there is a vacancy, if the vacancy is at the lower left of the ridge area, it is filled with the data of the edge of the ridge area above the corresponding vacancy; if the vacancy is at the upper right of the ridge area, it is filled with the data of the edge of the ridge area to the left of the vacancy.

Images