An online synthesis method of high dynamic range images for CMT welding scenes
By using triggers and FFHDR-Net modules to align and fuse welding images in CMT welding scenarios, the problem that ordinary industrial cameras cannot obtain high dynamic range images online is solved, and low-cost and efficient welding process information capture is achieved, improving welding Accuracy of quality inspection.
Patent Information
- Application Number
- CN202411197452.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-29
AI Technical Summary
Ordinary industrial cameras cannot obtain high dynamic range images online during CMT welding. The existing methods are complex and costly, making it difficult to meet the needs of real-time monitoring of the welding process.
A high dynamic range image online synthesis method for CMT welding scenes is designed. The trigger trigger camera is used to acquire welding images, and high dynamic range images are generated through the optimal synthetic frame count calculation module, SAM module and FFHDR-Net module, including multiple frames. Image alignment, feature fusion, and decoder reconstruction image resolution.
It realizes the use of ordinary industrial cameras to obtain high dynamic range images online, reduces equipment costs, and can simultaneously capture the melt pool, arc, wire extension length and weld information, improving the accuracy of welding quality detection and analysis.
Smart Images

Figure CN118714464B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an online synthesis method of high dynamic range images of CMT welding scenes. Background Art
[0002] Cold Metal Transfer (CMT) is an advanced welding technology. This technology can realize the monitoring of arc and wire feeding in CNC mode by digitally coordinating the droplet transfer and wire feeding movement. The wire feeding mechanism controls the pulsed wire feeding at a high frequency according to the instructions of the wire feeding control system, and adjusts the power supply output waveform at the same time. When the droplet and the molten pool are short-circuited, the droplet transfers in a state of no current. At this time, the arc is extinguished and the welding current is almost reduced to 0, thereby greatly reducing the welding heat input.
[0003] However, during the welding process, the illumination of the arc light is much greater than that of natural light. Therefore, when collecting welding images during the welding process, the arc is equivalent to the only light source. Due to the high brightness of the welding arc, ordinary industrial cameras cannot obtain information about the welding process and have a low dynamic range. Dynamic range refers to the range of brightness levels from darkest to brightest that an image can capture and display, or the ability of an image capture system to record details from darkest to brightest. During the welding process, a high dynamic range can obtain information such as the molten pool, arc, wire extension length, and weld or weld bead, which can effectively monitor the welding process and identify welding defects.
[0004] Patent publication number CN114757849A discloses an imaging method with a high dynamic range, which mainly obtains high dynamic images through a dedicated optical path system and a multi-image acquisition device combined with scale transformation, registration and parallel fusion technology. However, this method requires a precise optical path system and multiple image acquisition devices, and the calculation is complex and difficult to achieve online synthesis of high dynamic images. At the same time, the cost is high and it is not suitable for ordinary industrial cameras.
[0005] Patent publication number CN113240614B discloses a high dynamic image fusion method suitable for K-TIG welding ultra-strong arc scenes. It mainly generates high dynamic range images by fusing low dynamic range image sequences collected by ordinary industrial cameras through automatic exposure search, overall exposure ratio calculation, relative brightness recovery and tone remapping. However, this method is designed for K-TIG welding ultra-strong arc scenes and may not work well under other types of welding or lighting conditions. In addition, the method involves multiple steps, including random search exposure time, histogram reasoning, mask processing, morphological expansion and weighted averaging, which makes the algorithm more complicated and difficult to meet the requirements of online synthesis.
[0006] In order to obtain information about the CMT welding process in real time using a common industrial camera, it is necessary to design a method that can use a common industrial camera to obtain high dynamic range images online. Summary of the invention
[0007] The purpose of the present invention is to propose an online synthesis method of high dynamic range images of CMT welding scenes to solve the problem that ordinary industrial cameras cannot obtain high dynamic range images online during the CMT welding process due to their low dynamic range.
[0008] In order to achieve the above-mentioned purpose, the present invention provides an online synthesis method of high dynamic range images of CMT welding scenes, which uses a trigger to trigger the camera action to collect welding images during the welding process, and calculates the number of frames required for synthesizing the clearest high dynamic range image through the optimal synthesis frame number calculation module; the input multiple frames of welding images are aligned through the SAM module; the aligned features are integrated with global features through the FFHDR-Net module to generate a high dynamic range image that can simultaneously obtain the molten pool, arc, welding wire extension length and weld or weld bead information; the FFHDR-Net module includes multiple parallel convolution modules for extracting correlation features, the outputs of the multiple convolution modules are connected to the feature fusion network, and the feature fusion network is connected to the decoder.
[0009] Further preferably, the trigger consists of a current sensor, a signal conditioner and a trigger signal generator; the electrical signal output by the current sensor is transmitted to the signal conditioner, the signal conditioner converts the analog signal into a digital signal and outputs it to the trigger signal generator, and the trigger signal generator outputs a control signal to the camera.
[0010] Further preferably, the optimal synthesis frame number calculation module obtains the waveform of the periodic change of the current during the CMT welding process, determines the time for the current to change from small to large in each cycle, and calculates the number of frames required to synthesize the clearest high dynamic range image according to the periodic change. The calculation formula is as follows:
[0011] ;
[0012] in, is the optimal sampling frame number, is the change period between current peaks, is the time interval between two adjacent welding image frames, This is a rounding-up function, and its function is to ensure that the obtained number of frames is an integer value sufficient to capture the periodic change of the current.
[0013] Further preferably, the process of aligning the input multiple frames of welding images by the SAM module is as follows: the SAM module extracts features from different input welding images, and then divides the acquired feature maps into reference features and calculation features, and performs alignment modeling operations; the specific formula is as follows:
[0014] ;
[0015] in, represents the output features after normalization and skip connection, represents the normalization operation, is the reference feature, To calculate features;
[0016] ;
[0017] In the formula, Represents the features output by the SAM model, Cross represents cross alignment, represents the parameter matrix for linear transformation of the reference features, Represents the feature matrix for linear transformation of the calculated features, and T represents transpose.
[0018] Further preferably, during the alignment process, features that fail to align are deleted through a gating network.
[0019] Further preferably, the specific process of deleting the features of alignment failure through the gating network is: inputting the features into the gated recurrent unit, updating and transmitting the transmission information in the time step, determining the alignment failure features and the alignment success features; then calculating the importance update weights through the long short-term memory network; finally, after selectively updating the feature representation, the alignment success features are retained and strengthened, and the alignment failure features are weakened and suppressed.
[0020] Further preferably, the feature fusion network is composed of convolution and dense dilated convolution, the dense dilated convolution is composed of densely connected dilated convolution, and the ordinary convolution layer in the dense connection is replaced by the dilated convolution with the dilation coefficient; after the convolution operation, the input correlation feature will pass through several dense dilated convolutions and another convolution, and then return to each previous layer through a jump link as feedback for optimization, and finally output the fused global feature.
[0021] Further preferably, the decoder is composed of an upsampling layer, a convolution layer, and an activation function. The global features are input into the decoder to restore the image resolution to reconstruct the high dynamic range image, and the high-resolution shallow features are passed to the decoder through jump connections; the high-resolution shallow features and the low-resolution global features will first pass through the upsampling layer, then through the convolution layer, and finally through the activation function for fusion to generate the final high dynamic range image.
[0022] The present invention utilizes the fact that the current change in the CMT welding process is related to the arc brightness, and designs a trigger controlled by current to obtain welding images of different brightness. Then the obtained welding images of different brightness are input into a multi-frame image fusion network based on multi-exposure composed of the optimal synthesis frame number calculation module, SAM module, and FFHDR-Net network to synthesize a high dynamic range image. The present invention has the following beneficial effects:
[0023] 1. The present invention has lower requirements for image acquisition equipment. Compared with the traditional professional equipment for HDR image acquisition (such as high dynamic range cameras or special sensors), the ordinary industrial cameras used in the present invention are lower in cost, which reduces the equipment procurement and maintenance costs.
[0024] 2. The present invention uses an image synthesis method to generate online high dynamic range images that can simultaneously obtain the molten pool, arc, wire extension length, and weld or weld information. It can capture rich details and help improve the detection and analysis of welding quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flow chart of the present invention.
[0026] Figure 2 It is a schematic diagram of the trigger composition of the present invention.
[0027] Figure 3 This is the structural diagram of the FFHDR-Net module of the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is described clearly and completely below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0029] like Figure 1As shown, the present invention is an online synthesis method of high dynamic range images of CMT welding scenes. First, a trigger is used to trigger the camera action to collect welding images during the welding process; the number of frames required for synthesizing the clearest high dynamic range image is calculated through the optimal synthesis frame number calculation module; the input multiple frames of welding images are aligned through the SAM module to avoid the generation of artifacts; the aligned features are integrated with global features through the FFHDR-Net module to generate a high dynamic range image that can simultaneously obtain the molten pool, arc, wire extension length and weld or weld bead information.
[0030] The following describes each part in detail.
[0031] 1. Use triggers to trigger camera actions to collect welding images during the welding process: In order to control the amount of heat input during CMT welding, the input and stop of the welding wire are adjusted by the size of the current. When the current increases, the welding wire is transferred and the arc is ignited; when the current decreases, the welding wire stops transferring and the arc is extinguished. There is a corresponding relationship between the current size and the arc brightness. Therefore, the current change during CMT welding can be used as a control signal to control the camera shooting, and welding images with different brightness can be obtained. Triggers such as Figure 2 As shown in the figure, it is mainly composed of a current sensor, a signal conditioner and a trigger signal generator. Since the CMT welding process generates a large current, a Hall sensor is selected as the current sensor and installed near the current lead. When the current passes through, the Hall sensor will sense the electric field generated by the current, thereby generating a potential difference. Since the potential difference is proportional to the current passing through the conductor, the current can be obtained by measuring the potential difference. The calculation formula is as follows:
[0032] ;
[0033] in is the current passing through the conductor, is the output potential difference, For sensitivity.
[0034] After that, the obtained electrical signal needs to be input into the signal conditioner. Since the obtained electrical signal is an analog signal, if it is directly transmitted without processing, it is easy to introduce noise and cause distortion. Therefore, it is necessary to first convert the analog signal into a digital signal through a signal conditioner to more accurately capture the size and changes of the signal and enhance the clarity and reliability of the input signal. Finally, the signal is passed to the trigger signal generator to convert the electrical signal into a TTL signal. The TTL signal can be triggered quickly and accurately, and is suitable for camera systems. Finally, the signal is input into the camera for shooting.
[0035] 2. Optimal synthetic frame number calculation module: During the CMT welding process, the brightness and darkness change greatly. The synthetic high dynamic range image needs to cover a wide range of brightness from shadows to highlights. Therefore, it is necessary to determine the optimal frame number for the synthetic high dynamic range image to ensure that the final synthetic high dynamic range image can effectively capture the wide dynamic range and details in the scene. First, it is necessary to obtain the current change and draw a current change image. In a complete cycle, the current first rises to the highest value, the brightness gradually increases, and then remains unchanged for a period of time. Then the current begins to decrease, and the brightness continues to decrease until the arc is extinguished. By calculating the intervals between adjacent current peaks, a series of welding images with different currents are taken. By capturing welding images of corresponding brightness under different currents, the appropriate number of frames is calculated. The calculation formula is as follows:
[0036] ;
[0037] in is the optimal sampling frame number, is the change period between current peaks, is the time interval between two adjacent welding image frames, The purpose of this function is to round up to an integer and ensure that the number of frames obtained is an integer value sufficient to capture the periodic variation of the current.
[0038] 3. SAM module: During the CMT welding process, the camera is mounted on the welding gun and moves with the welding gun. Therefore, there will be misalignment in the multiple welding images taken. However, when synthesizing high dynamic range images, the pixels of multiple welding images need to correspond to the same focus and angle to ensure that the synthesized image will not have discontinuous and blurred areas. At the same time, image alignment can also ensure that the welding images are easier to match and fuse during the fusion process, reducing the complexity and computational burden in subsequent processing steps, which is particularly important for real-time generation of high dynamic range images. The commonly used alignment method is an alignment method based on image features, so the present invention designs a SAM module for image alignment. The operation process of the SAM module is as follows: First, the welding image with the most balanced exposure in the input welding image is selected as the reference image, and the remaining welding images are used as input images; then, the features used as reference and the features used for calculation are normalized. This step can effectively improve the generalization ability and increase the synthesis speed. Then the calculation result is added to the calculated features initially input to form a jump link. The calculation formula is as follows:
[0039] ;
[0040] in, represents the output features after normalization and skip connection, represents the normalization operation, is the reference feature, To calculate the features. Then perform SAM calculation on the obtained results to perform alignment. The formula is as follows:
[0041] ;
[0042] in, represents the features output by the SAM model, Cross represents cross alignment, T represents transposition, represents the parameter matrix for linear transformation of the reference features, Represents the parameter matrix for linear transformation of the computed features.
[0043] After that, it passes through the gating network, which is used to delete the features that failed to align during the alignment process. If it cannot be deleted in time, the wrong alignment features will participate in the synthesis of high dynamic range images, affecting the final effect. The specific process is: input the features into the gated recurrent unit, update and transmit the transmission information in the time step, determine the harmful information (i.e., alignment failure features) and correct information (i.e., alignment success features); then calculate the importance update weight through the long short-term memory network. Finally, after selectively updating the feature representation, the features with high importance (i.e., alignment success features) are retained and strengthened, and the features with low importance (i.e., alignment failure features) are weakened and suppressed. In this way, the wrong features are eliminated and the accuracy is improved. The calculation formula is as follows:
[0044] ;
[0045] in Features removed for alignment failures, is a nonlinear mapping, To adjust the parameters.
[0046] 4. FFHDR-Net module integrates global features: The FFHDR-Net module structure is as follows Figure 3 As shown in the figure, the aligned input features are first divided into different branches, and the correlation features are extracted by the convolution module respectively. The formula is as follows:
[0047] ;
[0048] in, Represents the correlation characteristics, represents bitwise multiplication, Represents the correlation coefficient.
[0049] Then, after passing through the feature fusion network, the information that is most capable of generating a high dynamic range image is selected from the correlation features output by different branches, such as foreground feature information or background feature information. After that, it is fused into a global feature. The feature fusion network is mainly composed of 3×3 convolutions and dense dilated convolutions. The dense dilated convolutions are composed of densely connected dilated convolutions, and the ordinary convolution layers in the dense connections are replaced by dilated convolutions with dilated convolutions with dilation coefficients. This is more conducive to obtaining features of larger pixel adjacent areas, and the features obtained by the dilated convolutions have a larger receptive field, which can better fuse global features and ensure that the integrity and continuity of the features are not destroyed during the fusion process. After the input features have undergone a convolution operation with a convolution kernel size of 3×3, they will pass through three dense dilated convolutions with a dilation coefficient of K and another 3×3 convolution, and then return to each previous layer through a jump link as feedback for optimization, and finally output the fused global features. The formula for fusing global features is as follows:
[0050] ;
[0051] in, Represents global features, Represents the correlation characteristics, Represents branch features.
[0052] After being integrated into global features, the global features are input into the decoder to restore the image resolution to reconstruct a high dynamic range image. The decoder is mainly composed of an upsampling layer, a convolution layer, and an activation function. Due to information loss during the forward propagation process, the input high-resolution information is missing, which affects the final quality. Therefore, the present invention passes the high-resolution shallow features to the decoder through a jump connection. The high-resolution shallow features and the low-resolution global features will first pass through the upsampling layer, then through two convolution layers, and finally through the activation function for fusion to restore the image's detail information and generate the final high dynamic range image. The formula is as follows:
[0053] ;
[0054] in, Represents the final output high dynamic range image, is the correction parameter, is a global feature, Represents an image compositing operation.
[0055] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be within the protection scope of the present invention.
Claims
1. A method for online synthesis of high dynamic range images of CMT welding scenes, characterized in that: A trigger is used to trigger the camera action to collect welding images during the welding process, and the number of frames required to synthesize the clearest high dynamic range image is calculated through the optimal synthesis frame number calculation module; the input multiple frames of welding images are aligned through the SAM module; the aligned features are integrated with global features through the FFHDR-Net module to generate a high dynamic range image that can simultaneously obtain the molten pool, arc, welding wire extension length and weld or weld bead information; the FFHDR-Net module includes multiple parallel convolution modules for extracting correlation features, the outputs of the multiple convolution modules are connected to the feature fusion network, and the feature fusion network is connected to the decoder; The optimal synthesis frame number calculation module obtains the waveform of the periodic change of the current during the CMT welding process, determines the time for the current to change from small to large in each cycle, and calculates the number of frames required to synthesize the clearest high dynamic range image according to the periodic change. The calculation formula is as follows: ; in, is the optimal sampling frame number, is the change period between current peaks, is the time interval between two adjacent welding image frames, is the ceiling rounding function; The process of aligning the input multiple frames of welding images through the SAM module is as follows: the SAM module extracts features from different input welding images, and then divides the acquired feature maps into reference features and calculation features for alignment modeling operations; the specific formula is as follows: ; in, represents the output features after normalization and skip connection, represents the normalization operation, is the reference feature, To calculate features; ; In the formula, Represents the features output by the SAM model, Cross represents cross alignment, represents the parameter matrix for linear transformation of the reference features, Represents the feature matrix for linear transformation of the calculated features, and T represents transpose; The feature fusion network is composed of convolution and dense dilated convolution. The dense dilated convolution is composed of densely connected dilated convolutions. The ordinary convolution layer in the dense connection is replaced by the dilated convolution with a dilation coefficient. After the convolution operation, the input correlation feature will pass through several dense dilated convolutions and another convolution, and then return to each previous layer as feedback for optimization through a skip link, and finally output the fused global feature. The decoder consists of an upsampling layer, a convolution layer, and an activation function. The global features are input into the decoder to restore the image resolution to reconstruct the high dynamic range image, and the high-resolution shallow features are passed to the decoder through jump connections; the high-resolution shallow features and the low-resolution global features will first pass through the upsampling layer, then the convolution layer, and finally be fused through the activation function to generate the final high dynamic range image.
2. The method for online synthesis of high dynamic range images of CMT welding scenes according to claim 1, characterized in that: The trigger is composed of a current sensor, a signal conditioner and a trigger signal generator; the electrical signal output by the current sensor is transmitted to the signal conditioner, the signal conditioner converts the analog signal into a digital signal and outputs it to the trigger signal generator, and the trigger signal generator outputs a control signal to the camera.
3. The method for online synthesis of high dynamic range images of CMT welding scenes according to claim 1, characterized in that: During the alignment process, features that fail to align are removed through a gating network.
4. The method for online synthesis of high dynamic range images of CMT welding scenes according to claim 3, characterized in that: The specific process of deleting the features of alignment failure through the gating network is as follows: input the features into the gated recurrent unit, update and transfer the transmission information in the time step, determine the alignment failure features and alignment success features; then calculate the importance update weights through the long short-term memory network; finally, after selectively updating the feature representation, the alignment success features are retained and strengthened, and the alignment failure features are weakened and suppressed.
