Brazing filler metal melting state detection method and device based on machine vision and electronic equipment

Abstract

The invention provides a brazing filler metal melting state detection method and device based on machine vision and electronic equipment, and relates to the technical field of brazing. The method comprises the following steps: acquiring brazing filler metal video data filtered by a lens of a camera, and sampling the brazing filler metal video data to obtain two frames of first brazing filler metal state images; performing optical flow field estimation on the two frames of first brazing filler metal state images by using a preset optical flow estimation network to obtain a forward optical flow field image; performing bidirectional consistency filtering on the forward optical flow field image to obtain an optical flow filtering image; classifying the optical flow filtering image by using a preset classification network to obtain a classification result; when the brazing filler metal is not melted, sampling the brazing filler metal video data according to a preset time interval to obtain two frames of second brazing filler metal state images, taking the second brazing filler metal state images as first brazing filler metal state images, and performing optical flow field estimation on the two frames of first brazing filler metal state images by using a preset optical flow estimation network to obtain a second brazing filler metal state image; by means of the method, the melting state of the brazing filler metal can be detected in real time.

Description

Technical Field

[0001] This application relates to the field of brazing technology, specifically to a method, apparatus, and electronic device for detecting the melting state of brazing filler metal based on machine vision. Background Technology

[0002] In brazing, dynamic monitoring of the solder's melting state is crucial for controlling weld quality. Currently, weld quality is primarily assessed using non-destructive testing techniques such as X-rays and ultrasound to evaluate the quality of post-weld components. However, these methods cannot monitor and intelligently classify the melting state in real time. Some techniques, drawing on experience from laser cladding, utilize unmelted powder particles as natural tracers, employing image segmentation algorithms and image velocimetry to quantify the flow field characteristics of the molten pool surface. However, image velocimetry is an offline analysis algorithm, limited by offline computational characteristics. Furthermore, it requires merging multiple images for calculation, resulting in significant computational delays and hindering accurate real-time feedback. Summary of the Invention

[0003] This application provides a machine vision-based method, apparatus, and electronic device for detecting the melting state of solder, which can detect the melting state of solder in real time and improve the soldering yield.

[0004] The technical solution of this application embodiment is as follows: In a first aspect, embodiments of this application provide a method for detecting the melting state of solder based on machine vision. The method includes: acquiring solder video data filtered by a camera lens, sampling the solder video data, and obtaining two frames of a first solder state image. Optical flow field estimation is performed on two frames of the first solder state image using a preset optical flow estimation network to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the trend of gray value changes in the first solder state image. The forward optical flow field image is subjected to bidirectional consistency filtering to obtain an optical flow filtered image; The optical flow filtered image is classified using a preset classification network to obtain a classification result, which indicates whether the solder has melted. If the classification result indicates that the solder has not melted, the solder video data is sampled at a preset time interval to obtain two frames of second solder state images. The second solder state images are used as the first solder state images. The step of using a preset optical flow estimation network to estimate the optical flow field of the two frames of the first solder state images to obtain a forward optical flow field image is executed.

[0005] In the above technical solution, firstly, the solder video data filtered by the camera lens is acquired. This lens filtering removes interference information. Acquiring the filtered solder video data reduces interference in subsequent optical flow estimation, providing data support for subsequent data processing. The solder video data is sampled to obtain two frames of the first solder state image. By sampling two frames, the continuity of the sampled images can be guaranteed, and the delay problem caused by the fusion of multiple images can be alleviated, enabling real-time detection. A preset optical flow estimation network is used to estimate the optical flow field of the two frames of the first solder state image to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the trend of changes in the grayscale values ​​of pixels in the first solder state image. By performing optical flow field estimation, the surface motion of the solder can be reflected. The algorithm employs a trend-based method to effectively detect the melting state. A bidirectional consistency filter is applied to the forward optical flow field image to obtain a filtered optical flow image. This filter verifies the accuracy of optical flow estimation, providing a foundation for subsequent accurate classification. A pre-defined classification network is used to classify the filtered optical flow image, yielding a classification result indicating whether the solder has melted. This method eliminates the need for offline processing and enables real-time detection of the solder fusion state. If the classification result indicates that the solder has not melted, the solder video data is sampled at pre-defined time intervals to obtain two frames of the second solder state image. These second solder state images are then used as the first solder state image. The algorithm then performs optical flow field estimation on these two frames using a pre-defined optical flow estimation network to obtain the forward optical flow field image. Through iterative double-frame detection, transient changes during the melting process are captured, reducing the latency caused by multiple images and achieving real-time detection and quality control of the solder melting state, thereby improving the welding yield.

[0006] In some embodiments of this application, performing bidirectional consistency filtering on the forward optical flow field image to obtain an optical flow filtered image includes: Reverse the timing of the two frames of the first solder state image to obtain two frames of reversed solder state image; Optical flow field estimation is performed on the two frames of the reverse solder state image using a preset optical flow estimation network to obtain the reverse optical flow field image; Error and confidence level processing is performed on the forward optical flow field image and the reverse optical flow field image to obtain the optical flow filtered image.

[0007] In some embodiments of this application, the step of performing error and confidence processing on the forward optical flow field image and the reverse optical flow field image to obtain the optical flow filtered image includes: Calculate the error between the pixels in the forward optical flow field image and the pixels in the reverse optical flow field image to obtain multiple error values; Confidence is calculated for each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence value corresponding to the pixel in the forward optical flow field image, wherein the optical flow amplitude is used to indicate the length of the optical flow vector; Each confidence value is compared with a preset confidence threshold. Pixels in the forward optical flow field image whose confidence value is greater than the confidence threshold are retained. Pixels in the forward optical flow field image whose confidence value is less than or equal to the confidence threshold are smoothed to obtain an optical flow filtered image.

[0008] In some embodiments of this application, the calculation of the error between pixels in the forward optical flow field image and pixels in the reverse optical flow field image yields multiple error values, including: The error between pixels in the forward optical flow field image and pixels in the reverse optical flow field image is calculated using a preset error formula, resulting in multiple error values. The preset error formula is expressed as follows: E abs (p)=||F forward (p)+W(F forward ,F backward (p)||2p represents a pixel, F forward () is the forward optical flow vector at point p, F backward Let W() be the reverse optical flow vector, and E be a reprojection function. abs (p) represents the error value corresponding to the pixel. The error value corresponding to the pixel is obtained by calculating the Euclidean distance between the forward flow vector corresponding to the pixel and the distorted forward flow vector.

[0009] In some embodiments of this application, the step of calculating the confidence level of each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence level value corresponding to the pixel in the forward optical flow field image includes: The confidence level of each error value and the corresponding preset optical flow amplitude are calculated using a preset confidence level formula to obtain the confidence level value of each pixel in the forward optical flow field image. The preset confidence level formula is expressed as follows: C(p) is the confidence value, p is a pixel in the forward optical flow field image, and σ abs and σ rel Both are adjustment parameters; the exp() function maps values ​​to the interval (0, 1], E abs (p) represents the error value corresponding to the pixel, E rel(p) represents the optical flow amplitude corresponding to the pixel.

[0010] In some embodiments of this application, the preset classification network includes a feature extraction module, a feature fusion module, and a classification module. The classification module includes a global average pooling layer, a feature expansion layer, an activation function layer, and a fully connected layer. The process of classifying the optical flow filtered image using a preset classification network to obtain classification results includes: The feature extraction layer is used to extract features from the optical flow filtered image to obtain multiple extracted features; The feature fusion module is used to fuse the extracted features to obtain fused features; The fused features are pooled using the global average pooling layer to obtain pooled features. The pooled features are expanded using the feature expansion layer to obtain one-dimensional feature data; The activation function layer is used to perform nonlinear processing on the one-dimensional feature data, and the fully connected layer is used to classify and map the nonlinearly processed data to output the classification result.

[0011] In some embodiments of this application, before acquiring the solder video data filtered by the camera lens, the method further includes: Determine whether the camera is working properly; if the camera is working properly, acquire the solder video data filtered by the camera lens. The camera has a bandpass filter in front of its lens, which is used to filter out the flame during welding.

[0012] Secondly, embodiments of this application provide a machine vision-based solder melting state detection device, the device comprising: The data acquisition module is used to acquire the brazing filler video data filtered by the camera lens, and to sample the brazing filler video data to obtain two frames of the first brazing filler state image. The estimation calculation module is used to perform optical flow field estimation on two frames of the first solder state image using a preset optical flow estimation network to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the trend of the gray value of the pixel in the first solder state image. The filtering module is used to perform bidirectional consistency filtering on the forward optical flow field image to obtain an optical flow filtered image; the classification module is used to classify the optical flow filtered image using a preset classification network to obtain a classification result, wherein the classification result indicates whether the solder has melted. The adjustment module is used to sample the solder video data at a preset time interval when the classification result indicates that the solder has not melted, to obtain two frames of second solder state images, and to use the second solder state images as the first solder state images, and to perform the step of using a preset optical flow estimation network to estimate the optical flow field of the two frames of the first solder state images to obtain a forward optical flow field image.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a user interface, a communication bus, and a network interface. The processor, the memory, the user interface, and the network interface are respectively connected to the communication bus. The memory is used to store instructions. The user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method described in any one of the first aspects.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, perform the method described in any one of the methods provided in the first aspect above.

[0015] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By first acquiring the solder video data filtered by the camera lens, interference information can be removed. Acquiring the filtered solder video data reduces interference in subsequent optical flow estimation, providing data support for subsequent data processing. The solder video data is sampled to obtain two frames of the first solder state image. Dual-frame sampling ensures the continuity of the sampled images and alleviates the latency problem caused by multiple image fusion, enabling real-time detection. A preset optical flow estimation network is used to estimate the optical flow field of the two frames of the first solder state image, obtaining a forward optical flow field image. The optical flow field estimation process characterizes the trend of grayscale value changes in pixels in the first solder state image. Optical flow field estimation reflects the surface motion trend of the solder. The method employs a pre-defined optical flow field network to effectively detect the melting state. A bidirectional consistency filter is applied to the forward optical flow field image to obtain a filtered optical flow image. This filter allows for verification of the accuracy of optical flow estimation, providing a foundation for subsequent accurate classification. The filtered optical flow image is then classified using a pre-defined classification network to obtain classification results indicating whether the solder has melted. This eliminates the need for offline processing and enables real-time detection of the solder fusion state. If the classification result indicates that the solder has not melted, the solder video data is sampled at pre-defined time intervals to obtain two frames of second solder state images. These second solder state images are used as the first solder state images, and the pre-defined optical flow estimation network is used to estimate the optical flow field of the two frames to obtain the forward optical flow field image. Through iterative double-frame detection, transient changes during the melting process are captured, reducing the latency caused by multiple images and achieving real-time detection and quality control of the solder melting state, thus improving the welding yield. Therefore, this method effectively solves the computational latency caused by offline calculations and multiple image processing in related technologies, preventing real-time detection.

[0016] 2. By calculating the forward and reverse optical flow field images, utilizing the error between the reverse and forward optical flow field images, and determining the confidence level, the forward optical flow field image can be filtered, which can suppress optical flow noise caused by excessive sampling interval.

[0017] 3. By redesigning the classification module, the model can be made lightweight and computational complexity can be reduced, thus meeting the requirements of real-time detection.

[0018] 4. By setting a bandpass filter, the flame during welding can be filtered out, improving data reliability. Attached Figure Description

[0019] Figure 1 This is a schematic flowchart of a machine vision-based solder melting state detection method provided in one embodiment of this application; Figure 2 This is a schematic diagram of a data acquisition structure provided in one embodiment of this application; Figure 3 This is a schematic diagram of the classification network structure of a machine vision-based solder melting state detection method provided in one embodiment of this application; Figure 4 This is a schematic diagram of the solder melting state of a solder melting state detection method based on machine vision provided in one embodiment of this application; Figure 5 This is a schematic diagram of the melting state result of a machine vision-based solder melting state detection method provided in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a machine vision-based solder melting state detection device provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0021] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0022] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0023] In related technologies, current methods for inspecting welding quality include non-destructive testing of components using X-rays and ultrasound. However, these methods cannot monitor and intelligently classify the melting state in real time. Currently, the focus of machine vision applications in industry is mainly on defect detection after welding, which also lacks real-time capability. For laser welding, unmelted powder in laser cladding is used as natural tracer particles, processed through the following modules: Image acquisition module: high-speed camera captures dynamic image sequences (multiple images) of the molten pool; Segmentation module: extracts powder targets, resolving interference from molten pool reflections and oscillations; Analysis module: responsible for calculating particle displacement. While this approach successfully characterizes the non-uniform flow field on the molten pool surface, its shortcomings are also significant. The data processed by the algorithm depends on specific processes, is only applicable to laser cladding welding, and heavily relies on unmelted powder as tracer particles. Since unmelted powder does not exist in brazing, this approach is not suitable.

[0024] Based on this, embodiments of this application provide a machine vision-based method, apparatus, electronic device, and readable storage medium for detecting the melting state of solder. The machine vision-based method first acquires solder video data filtered by a camera lens. This lens filtering removes interference information, and the filtered solder video data reduces interference in subsequent optical flow estimation, providing data support for subsequent data processing. The solder video data is sampled to obtain two frames of the first solder state image. This dual-frame sampling ensures the continuity of the sampled images and alleviates the delay caused by multiple image fusion, enabling real-time detection. A preset optical flow estimation network is used to estimate the optical flow field of the two frames of the first solder state image, obtaining a forward optical flow field image. The optical flow field estimation process characterizes the grayscale values ​​of pixels in the first solder state image. The trend of change in the solder surface can be reflected by optical flow field estimation, enabling effective detection of the melting state. A bidirectional consistency filter is applied to the forward optical flow field image to obtain a filtered optical flow image. This filter verifies the accuracy of the optical flow estimation, providing a foundation for accurate classification. A pre-defined classification network is used to classify the filtered optical flow image, yielding a classification result indicating whether the solder has melted. This eliminates the need for offline processing and allows for real-time detection of the solder fusion state. If the classification result indicates that the solder has not melted, the solder video data is sampled at pre-defined time intervals to obtain two frames of the second solder state image. These second solder state images are used as the first solder state image, and the pre-defined optical flow estimation network is used to estimate the optical flow field of the two frames to obtain the forward optical flow field image. Through iterative double-frame detection, transient changes during the melting process are captured, reducing the delay caused by multiple images and achieving real-time detection and quality control of the solder melting state, thus improving the welding yield.

[0025] It should be noted that this machine vision-based brazing filler metal melting state detection method is used for detecting the melting state of brazing filler metal in industrial welding, assisting in the monitoring of the welding process, preventing over-welding, controlling welding quality, and improving the weld yield. It can also be applied to the detection of melting states in metal smelting, plastic injection molding, and glass melting.

[0026] The technical solutions provided in the embodiments of this application will be further described below with reference to the accompanying drawings.

[0027] Reference Figure 1 , Figure 1 This is a schematic flowchart of a machine vision-based solder melting state detection method provided in an embodiment of this application. The machine vision-based solder melting state detection method is applied to a machine vision-based solder melting state detection device and is executed by a processor in an electronic device or a readable storage medium. The machine vision-based solder melting state detection method includes steps S100, S200, S300, S400, and S500.

[0028] Step S100: Obtain the brazing filler video data filtered by the camera lens, sample the brazing filler video data, and obtain two frames of the first brazing filler state image.

[0029] In one embodiment, before acquiring the solder video data filtered by the camera lens, the machine vision-based solder melting state detection method further includes, but is not limited to: determining whether the camera is working properly, and if the camera is working properly, acquiring the solder video data filtered by the camera lens; wherein, a bandpass filter is provided in front of the camera lens, and the bandpass filter lens is used to filter out the flame during welding.

[0030] Specifically, a bandpass filter is installed in front of the camera lens to filter out the flame during welding. The bandpass filter can be 400-480nm or 380-460nm band filtering. An LED light can also be installed in front of the camera lens to supplement illumination and reduce poor image quality caused by poor lighting. When the camera is operating normally, it can acquire normal image data and transmit the weld metal video data to the data processing module. The data processing module obtains the weld metal video data filtered by the camera lens, reducing interference in subsequent optical flow estimation and providing data support for subsequent data processing.

[0031] It should be noted that if the camera is not working properly, it means that normal image acquisition is impossible, and therefore subsequent solder melting analysis is also impossible, thus ending the process.

[0032] In another embodiment, the lens-filtered brazing video data acquired by the camera is specifically captured at 25 frames per second. This brazing video data is then transmitted to provide data support for subsequent data processing. The brazing video data is then sampled to obtain two frames of the first brazing state image. Specifically, following the order in the brazing video data, the brazing state image is sampled at approximately 2 frames per second, resulting in two frames of the first brazing state image. This dual-frame sampling ensures the continuity of the sampled images and alleviates the latency problem caused by the fusion of multiple images, enabling real-time detection.

[0033] It should be noted that, in this embodiment, after capturing the solder video at the camera end, two frames of the first solder status image can be directly sampled at 2 frames per second, and then transmitted to the data processing module in the computer for subsequent data processing. By sampling two frames of images at the camera end for transmission, the latency caused by data transmission and the latency caused by the fusion of multiple images can be effectively avoided, enabling effective real-time detection.

[0034] like Figure 2 As shown, this machine vision-based solder melting state detection method is executed in a computer connected to an industrial camera equipped with a zoom industrial lens. A 400-480nm bandpass filter is placed in front of the zoom industrial lens to filter the flame. Interference is removed by filtering through the camera lens. Then, the camera samples two frames of the first solder state image and transmits them to the computer for subsequent data processing to perform real-time solder melting state detection and assist in welding quality control.

[0035] Step S200: Optical flow field estimation is performed on two frames of the first solder state image using a preset optical flow estimation network to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the trend of changes in the gray value of the pixels in the first solder state image.

[0036] In one embodiment, the preset optical flow estimation network is a SpyNet network. This SpyNet network processes the first solder state image through downsampling and pyramid processing to achieve optical flow field estimation, obtaining a forward optical flow field image. The optical flow field estimation process characterizes the trend of grayscale value changes in pixels in the first solder state image. An optical flow distribution map is obtained from two frames, reflecting the optical flow distribution and thus the trend of grayscale value changes in pixels. The optical flow distribution map is transformed into an RGB image (pass) that can be processed by a classification network, resulting in the optical flow field image. The optical flow distribution map uses different color systems and depths to represent the magnitude and direction of pixel velocity. If the solder has not yet melted, its fluidity is low, and the optical flow distribution map tends to be white and pale. If the solder has melted, the flow change between the two frames is large, and the optical flow distribution map tends to be more colorful. The optical flow estimation network can also be FlowNet, PWC-Net, etc.

[0037] It should be noted that the specific processing procedure of the SpyNet network is as follows: First, the input data consists of two frames of the first solder state image, represented as I1 and I2 respectively. The two frames of the first solder state image are downsampled, and the downsampled solder image is downsampled again to form a three-layer pyramid (represented as the first layer, the second layer, and the third layer). The data at the top layer (the third layer) of the pyramid is input into the convolutional neural network for processing, and a coarse, low-resolution optical flow field F3 is initially estimated. Since the optical flow field F3 is at the top layer, the two downsampled images are relatively blurry, but the main direction of motion can be captured. The estimated optical flow field F3 is upsampled, and the upsampled optical flow field is denoted as F33. F33 is matched with the downsampled image data of the second layer, and F33 is used to correct the optical flow estimation of the second layer. Specifically, the next frame downsampled image in the second layer is distorted in the opposite direction of F33 to generate a distorted image. The distorted image, the previous frame downsampled image in the second layer, and the upsampled optical flow field are input into a convolutional neural network for residual flow calculation to obtain the residual. The residual is then used to update the upsampled optical flow field. Specifically, the update process is to add the upsampled optical flow field to the residual to obtain the updated optical flow field.

[0038] Upsampling is performed using the updated optical flow field, and the first layer data is processed according to the steps described above to obtain the forward optical flow field. This forward optical flow field is then converted to RGB format using color encoding to obtain the forward optical flow field image. This forward optical flow field image can characterize the changing trends of pixels, facilitating accurate classification processing in subsequent steps.

[0039] Step S300: Perform bidirectional consistency filtering on the forward optical flow field image to obtain the optical flow filtered image.

[0040] In one embodiment, a bidirectional consistency filter is performed on the forward optical flow field image to obtain an optical flow filtered image, including but not limited to the following steps: Step S310: Reverse the timing of the two first solder state images to obtain two reversed solder state images.

[0041] In some possible embodiments of this application, the order of two frames of the first solder state image is swapped according to time sequence. For example, the original two frames of the first solder state image are frame 12 and frame 24, with smaller frame numbers indicating earlier timestamps. During flow field estimation, the pixel change trend follows the trend from frame 12 to frame 24. After time reversal, the frame numbers are swapped, and the solder state image of frame 12 becomes the solder state image of frame 24, i.e., two inverted solder state images. During flow field estimation, the pixel change trend follows the trend from frame 12 to frame 24, which is actually the trend from frame 24 to frame 12. By performing time reversal, two inverted solder state images are obtained, which can then be used to determine the fusion state shown by the two first solder state images for subsequent classification calculations.

[0042] Step S320: Optical flow field estimation is performed on two frames of reverse solder state images using a preset optical flow estimation network to obtain reverse optical flow field images.

[0043] In some possible embodiments of this application, a preset optical flow estimation network, SpyNet, is used. This SpyNet network processes two frames of inverted solder state images through downsampling and pyramid processing to achieve optical flow field estimation, resulting in a reverse optical flow field image. The optical flow field estimation process is used to characterize the trend of grayscale value changes in the pixels of the first solder state image. It should be noted that the process of using the preset optical flow estimation network to perform optical flow field estimation on two frames of inverted solder state images is similar to the process of using the preset optical flow estimation network to perform optical flow field estimation on two frames of the first solder state image; the aforementioned processing steps can be referred to, and will not be repeated here.

[0044] Step S330: Error and confidence processing are performed on the forward optical flow field image and the reverse optical flow field image to obtain the optical flow filtered image.

[0045] In some possible embodiments of this application, based on the forward optical flow field image and the reverse optical flow field image of the above steps, the forward optical flow field image and the reverse optical flow field image are first processed for error, and then confidence is calculated. Based on the calculated confidence, the forward optical flow field image is processed to obtain an optical flow filtered image. The accuracy of optical flow estimation can be checked through filtering, which provides a basis for subsequent accurate classification.

[0046] In one embodiment, error and confidence level processing is performed on the forward and reverse optical flow field images to obtain an optical flow filtered image, including but not limited to the following steps: Step S331: Calculate the error between the pixels in the forward optical flow field image and the pixels in the reverse optical flow field image to obtain multiple error values.

[0047] In some possible embodiments of this application, for each pixel in the forward optical flow field image and the corresponding pixel in the reverse optical flow field image, the error between the pixels is calculated, resulting in multiple error values. Since the optical flow field image reflects the optical flow distribution map of the pixels, the velocity magnitude and direction of each pixel constitute the optical flow vector in the optical flow distribution map. The error value corresponding to each pixel can be obtained through the above error formula.

[0048] Specifically, the error between pixels in the forward optical flow field image and pixels in the reverse optical flow field image is calculated to obtain multiple error values, including but not limited to: using a preset error formula to calculate the error between pixels in the forward optical flow field image and pixels in the reverse optical flow field image to obtain multiple error values; The preset error formula is expressed as follows: E abs (p)=||F forward (p)+W(F forward ,F backward (p)||2p represents a pixel, F forward () is the forward optical flow vector at point p, F backward Let W() be the reverse optical flow vector, and E be a reprojection function. abs (p) represents the error value corresponding to the pixel. The error value corresponding to the pixel is obtained by calculating the Euclidean distance between the forward flow vector corresponding to the pixel and the distorted forward flow vector.

[0049] It should be noted that the W() reprojection function uses the reverse optical flow vector to distort or map the forward optical flow vector. That is, it maps the forward optical flow vector from the coordinate system of the next frame to the original coordinate system of the previous frame using the reverse optical flow. This mapping can be done using bilinear interpolation. After distortion, a motion opposite to the original optical flow field is obtained, which is -F. forward In the above formula, the forward optical flow vector of each pixel is added to the distorted optical flow vector. Since they are opposites, their sum should be 0. Therefore, a smaller error value is better, indicating a more accurate optical flow estimation. This formula allows us to calculate the error value for each pixel, which is then used for subsequent confidence level calculations.

[0050] Step S332: Calculate the confidence level of each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence level value of the pixel in the forward optical flow field image, wherein the optical flow amplitude is used to indicate the length of the optical flow vector.

[0051] In one embodiment, the preset optical flow amplitude is used to indicate the length of the optical flow vector. It can be calculated by the direction and magnitude of the optical flow velocity. The direction and magnitude are squared respectively, and the squared values ​​are summed and then taken to obtain the optical flow amplitude, which facilitates subsequent confidence calculation based on the optical flow amplitude.

[0052] Specifically, the confidence level of each error value and the preset optical flow amplitude corresponding to the error value is calculated to obtain the confidence level value corresponding to the pixel in the forward optical flow field image. This includes, but is not limited to: using a preset confidence level formula to calculate the confidence level of each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence level value corresponding to the pixel in the forward optical flow field image. The preset confidence level formula is expressed as follows: C(p) is the confidence score, p is a pixel in the forward optical flow field image, and σ abs and σ rel Both are adjustment parameters; the exp() function maps values ​​to the (0,1] interval, E abs (p) represents the error value corresponding to the pixel, E rel (p) represents the optical flow amplitude corresponding to the pixel.

[0053] In some possible embodiments of this application, based on the error value of each pixel obtained in step S331, the confidence level is calculated by comparing the error value with the optical flow amplitude. A larger error value corresponds to a smaller optical flow amplitude and a lower confidence level, i.e., lower reliability; conversely, a smaller error value corresponds to a larger optical flow amplitude and a higher confidence level, i.e., higher reliability. Adjusting parameter σ... abs The value can be 1, adjusting the parameter σ. rel The value is set to 0.3, but other settings can be made. Adjusting these parameters changes the error value and optical flow amplitude value, ensuring that a smaller error corresponds to a larger optical flow amplitude value and a higher confidence level, indicating a more reliable optical flow estimation result for the current pixel. Mapping the error value and optical flow amplitude value to the range of 0 to 1 (normalization) reduces computational complexity. Multiplying the normalized error value by the normalized optical flow amplitude further demonstrates that a smaller error value corresponds to a larger optical flow amplitude and a higher confidence level.

[0054] It should be noted that E rel(p) represents the optical flow amplitude corresponding to the pixel, calculated using the magnitude and direction of the optical flow vector, reflecting the pixel variation trend in optical flow field estimation. Multiplying the above normalized values ​​instead of adding them reflects that the influence of amplitude and error on confidence is independent and complementary. The confidence is high only when both perform well, which aligns with the final confidence calculation and avoids problems such as overestimation of confidence caused by addition.

[0055] Step S333: Compare each confidence value with a preset confidence threshold, retain the pixels in the forward optical flow field image corresponding to the confidence value greater than the confidence threshold, and smooth the pixels in the forward optical flow field image corresponding to the confidence value less than or equal to the confidence threshold to obtain an optical flow filtered image.

[0056] In some possible embodiments of this application, a confidence value is obtained according to step S332, and the higher the confidence value, the more reliable the estimation. The confidence threshold reflects which pixels in the forward optical flow field image can be retained, i.e., the pixels with accurate estimation and those that reflect the melting state. This confidence threshold is obtained through experimentation and statistical analysis using a large amount of historical data.

[0057] The confidence value corresponding to each pixel is compared with a preset confidence threshold. The difference between the confidence value and the confidence threshold is used to determine the magnitude of the confidence value and the confidence threshold. Pixels in the forward optical flow field image with confidence values ​​greater than the confidence threshold are retained, indicating that these pixels have high reliability. Pixels in the forward optical flow field image with confidence values ​​less than or equal to the confidence threshold are smoothed, indicating that these pixels have low reliability and may be subject to interference from temperature, strong light, etc. Through the above processing, an optical flow filtered image is obtained.

[0058] The smoothing process can involve interpolating high-confidence regions in the forward optical flow field image to fill low-confidence regions, using methods such as Poisson interpolation or bilinear interpolation. This smoothing process eliminates abnormal optical flow values ​​caused by interference, thereby improving the accuracy of subsequent classification.

[0059] Step S400: Classify the optical flow filtered image using a preset classification network to obtain the classification result.

[0060] In one embodiment, the preset classification network structure is a lightweight network, such as... Figure 3As shown, the input to the preset classification network is an optically flow filtered image. The classification network structure includes a feature extraction module, a feature fusion module, and a classification module. The feature extraction module includes multiple convolutional layers, C3K2 layers, SPPF layers, and C2PSA layers. The C3K2 layer is the feature extraction module in the YOLOv11 model. It preserves shallow features by passing the path, extracts deep features by processing the path, and finally fuses features from different levels, reducing computational complexity and enabling real-time detection. The SPPF layer improves detection accuracy by segmenting the image into superpixel regions with similar color and texture features. The C2PSA layer is a composite structure combining channel attention and spatial attention, designed to enhance the model's ability to learn important information from feature maps, improving multi-scale feature extraction capabilities and thus increasing detection accuracy.

[0061] In the feature extraction module, convolutional layers and C3K2 layers are spaced apart. The input optical flow filtered image undergoes feature extraction sequentially through convolutional layers, C3K2 layers, and so on (the features extracted in this layer are used for subsequent feature concatenation to avoid gradient vanishing). Finally, features are extracted through an SPPF layer and then further processed by a C2PSA layer. Multiple extracted features are obtained by utilizing the feature extraction layers to extract features from the optical flow filtered image, as shown in the reference diagram. Figure 3 The extracted features include features from the C2PSA output and features from one of the C3K2 layers, which is a C3K2 layer located in the middle of a plurality of C3K2 layers.

[0062] Then, a feature fusion module is used to fuse the extracted features. This module includes an upsampling layer, a feature connection layer, a C3K2 layer, and a convolutional layer. The fusion process first upsamples the extracted features output from the C2PSA layer in the feature extraction module. Then, the sampled features are concatenated with the extracted features output from one of the C3K2 layers. This feature concatenation increases the representational power of the features and avoids the gradient vanishing problem that occurs when the number of network layers increases. The concatenated features are then sequentially processed through a C3K2 layer for feature extraction, an upsampling layer for upsampling, a feature concatenation layer for feature concatenation, a C3K2 layer for feature extraction, a convolutional layer for feature extraction, a feature concatenation layer for feature concatenation (combining the features output from the convolutional layer with the features extracted by the first C3K2 layer in the fusion module), a C3K2 layer for feature extraction, a convolutional layer for feature extraction, and a feature connection layer for feature concatenation (combining the features output from the previous convolutional layer connected to this feature connection layer with the extracted features output from the feature extraction module). Finally, after C3K2 layer feature extraction, the fused features are output for subsequent classification by the classification module.

[0063] In some possible embodiments of this application, the classification module includes a global average pooling layer, a feature expansion layer, an activation function layer, and a fully connected layer. The global average pooling layer is used to pool the fused features to obtain pooled features; the feature expansion layer is used to expand the pooled features to obtain one-dimensional feature data; the activation function layer is used to perform nonlinear processing on the one-dimensional feature data; and the fully connected layer is used to perform classification mapping on the nonlinearly processed data to output the classification result. The classification result indicates whether the solder has melted.

[0064] The classification result indicates whether the solder has melted, with two categories: melted and unmelted. The global average pooling layer averages the feature pixels, the feature unfolding layer flattens the feature vectors to obtain one-dimensional feature data, the activation function layer uses the sigmoid function, and the fully connected layer extracts features and outputs the binary classification result. This classification network is lightweight and does not require offline processing, enabling real-time detection of the solder fusion status.

[0065] It should be noted that the preset classification network is a pre-trained network. The image data and the corresponding optical flow field image collected by the industrial camera are manually labeled and then input into the classification network for training to obtain the pre-trained preset classification network. This classification network has high accuracy in classifying the melting state of the solder.

[0066] The above-described network structures are merely examples. Depending on the computational power and accuracy requirements, other suitable network structures can be used, such as MobileNet, ShuffleNet, or other lightweight networks, or other CNN networks that have been pruned and quantized to achieve lightweight networks that do not require offline image processing.

[0067] In step S500, if the classification result indicates that the solder has not melted, the solder video data is sampled at a preset time interval to obtain two frames of the second solder state image, and the second solder state image is used as the first solder state image.

[0068] In one embodiment, the second solder state image is used as the first solder state image, and then the step of using a preset optical flow estimation network to estimate the optical flow field of the two frames of the first solder state image to obtain the forward optical flow field image is executed cyclically.

[0069] In one embodiment, when the classification result indicates that the solder has not melted, the fusion judgment may be inaccurate because the sampled image may be from the very beginning of the welding process. Therefore, the solder video data is sampled at preset time intervals, such as 2 frames per second, to obtain two second solder state images. Since two frames that are too close together may not produce a clear change from unmelted to melted, increasing the sampling interval reduces computational costs and improves efficiency. Subsequent processing also samples two images according to the above rules, instead of performing image fusion processing on all images to determine whether it has melted, thus reducing computational complexity and achieving real-time solder melting state detection. Then, the second solder state image is used as the first solder state image. The optical flow field of the two first solder state images is estimated again using a preset optical flow estimation network to obtain a forward optical flow field image. A bidirectional consistency filter is then applied to the forward optical flow field image to obtain a filtered optical flow image. A preset classification network is used to classify the filtered optical flow image to obtain a classification result. The classification result indicates whether the solder has melted. The processing procedure of the above steps is similar to that of steps S200 to S400, and will not be elaborated here. It should be noted that if the classification result of the last two frames of the final sampled solder video data indicates that the solder has not melted, then the welding quality is unqualified, thereby controlling the welding quality and improving the welding yield.

[0070] It should be noted that if the classification result indicates melting, it means that the soldering process is complete and the above process ends.

[0071] like Figure 4 As shown, Figure 4 Images (a), (b), (c), and (d) illustrate the use of optical flow field images to determine whether melting has occurred. Before the solder melts, the overall shape of the two images shows little change. However, in the frames where the solder melts, the shape changes drastically and flow occurs. Therefore, optical flow field analysis can be used to determine whether melting has occurred. (a) shows the original state of the unmelted solder, and (c) shows the melted state. Optical flow estimation is performed on the two images of the unmelted solder to obtain (b). Optical flow estimation is performed on the unmelted and melted images to obtain (d). Based on the color distribution representing the direction and magnitude of flow velocity in the optical flow field images, it can be seen that the optical flow field is darker and more vibrant in the melted state, while it is more pale in the unmelted state.

[0072] For example, tests were also conducted in a copper pipe brazing scenario. The results showed a significant improvement in the accuracy of judging the melting state of the brazing filler metal, and a significant improvement in real-time performance compared to existing technologies, meeting the requirements of industrial real-time control. Offline analysis methods suffer delays exceeding 2 seconds, which cannot meet the requirements for real-time detection.

[0073] like Figure 5As shown in the figure, the optical flow field images obtained at different frame intervals in this embodiment of the application can reflect the solder fusion state. The image labeled "frame" is a sample of two frames representing the first solder state from solder video data captured by an industrial camera. The numbers indicate the frame number, with larger intervals representing larger intervals between frames. "Flows" represents the forward optical flow field image, and its hue and intensity indicate the direction and magnitude of pixel flow velocity. It can be seen that the optical flow field images obtained between frames with large intervals and obvious melting tend to have bright and rich colors, while the frames between frames that have not yet melted show obvious white or light-colored areas.

[0074] like Figure 6 As shown in the figure, this application embodiment provides a machine vision-based solder melting state detection device 100. This machine vision-based solder melting state detection device 100 acquires solder video data filtered by a camera lens. This lens filtering removes interference information, and the filtered solder video data reduces interference in subsequent optical flow estimation, providing data support for subsequent data processing. The solder video data is sampled to obtain two frames of first solder state images. Double-frame sampling ensures the continuity of the sampled images and alleviates the delay caused by multiple image fusion, enabling real-time detection. Then, the estimation calculation module 120 uses a preset optical flow estimation network to perform optical flow field estimation on the two frames of first solder state images to obtain a forward optical flow field image. The optical flow field estimation process characterizes the trend of grayscale value changes in pixels in the first solder state image. Through optical flow field estimation... This process reflects the surface movement trend of the solder, enabling effective detection of the melting state. The filtering module 130 then performs bidirectional consistency filtering on the forward optical flow field image to obtain a filtered optical flow image. This filtering allows for verification of the accuracy of optical flow estimation, providing a foundation for subsequent accurate classification. The classification module 140 uses a preset classification network to classify the filtered optical flow image, obtaining a classification result indicating whether the solder has melted. This process eliminates the need for offline processing and enables real-time detection of the solder fusion state. Finally, the adjustment module 150, when the classification result indicates that the solder has not melted, samples the solder video data at preset time intervals to obtain two frames of the second solder state image. These second solder state images are used as the first solder state image, and the preset optical flow estimation network is used to estimate the optical flow field of the two frames to obtain the forward optical flow field image. Through iterative double-frame detection, transient changes during the melting process are captured, reducing the delay caused by multiple images and achieving real-time detection and quality control of the solder melting state, thus improving the welding yield.

[0075] It should be noted that the data acquisition module 110 is connected to the estimation calculation module 120, the estimation calculation module 120 is connected to the filtering processing module 130, the filtering processing module 130 is connected to the classification processing module 140, and the classification processing module 140 is connected to the adjustment module 150. The above-mentioned machine vision-based solder melting state detection method is applied to a machine vision-based solder melting state detection device 100. The machine vision-based solder melting state detection device 100 acquires solder video data filtered by a camera lens. This lens filtering removes interference information. Acquiring filtered solder video data reduces interference in subsequent optical flow estimation, providing data support for subsequent data processing. The solder video data is sampled to obtain two frames of the first solder state image. Double-frame sampling ensures the continuity of the sampled images and alleviates the delay caused by multiple image fusion, enabling real-time detection. An optical flow estimation network is used to estimate the optical flow field of the two frames of the first solder state image to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the changes in grayscale values ​​of pixels in the first solder state image. The changing trend, reflected by optical flow field estimation, can indicate the surface movement trend of the solder, enabling effective detection of the melting state. A bidirectional consistency filter is applied to the forward optical flow field image to obtain a filtered optical flow image. This filter verifies the accuracy of the optical flow estimation, providing a foundation for subsequent accurate classification. A pre-defined classification network is used to classify the filtered optical flow image, yielding classification results indicating whether the solder has melted. This eliminates the need for offline processing and allows for real-time detection of the solder fusion state. If the classification result indicates that the solder has not melted, the solder video data is sampled at pre-defined time intervals to obtain two frames of the second solder state image. These second solder state images are used as the first solder state image, and the pre-defined optical flow estimation network is used to estimate the optical flow field of the two frames to obtain the forward optical flow field image. Through iterative double-frame detection, transient changes during the melting process are captured, reducing the latency caused by multiple images and achieving real-time detection and quality control of the solder melting state, thus improving the welding yield.

[0076] It should also be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0077] This application also discloses an electronic device. (See reference...) Figure 7 , Figure 7This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0078] The communication bus 502 is used to enable communication between these components.

[0079] The user interface 503 may include a display screen and a camera. Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.

[0080] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0081] The processor 501 may include one or more processing cores. The processor 501 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0082] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 7 The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for detecting the melting state of solder based on machine vision.

[0083] exist Figure 7 In the illustrated electronic device 500, the user interface 503 is mainly used to provide an input interface for the user and acquire user input data; while the processor 501 can be used to call an application program stored in the memory 505 for detecting the melting state of solder based on machine vision. When executed by one or more processors 501, the electronic device 500 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0084] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0085] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0089] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0090] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for detecting the melting state of solder based on machine vision, characterized in that, The method includes: Acquire the brazing filler video data filtered by the camera lens, sample the brazing filler video data, and obtain two frames of the first brazing filler state image; Optical flow field estimation is performed on two frames of the first solder state image using a preset optical flow estimation network to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the trend of gray value changes in the first solder state image. The forward optical flow field image is subjected to bidirectional consistency filtering to obtain an optical flow filtered image; The optical flow filtered image is classified using a preset classification network to obtain a classification result, which indicates whether the solder has melted. If the classification result indicates that the solder has not melted, the solder video data is sampled at a preset time interval to obtain two frames of second solder state images. The second solder state images are used as the first solder state images. The step of using a preset optical flow estimation network to estimate the optical flow field of the two frames of the first solder state images to obtain a forward optical flow field image is executed.

2. The method according to claim 1, characterized in that, The step of performing bidirectional consistency filtering on the forward optical flow field image to obtain an optical flow filtered image includes: The two frames of the first solder state image are time-reversed to obtain two frames of reversed solder state image; Optical flow field estimation is performed on the two frames of the reverse solder state image using a preset optical flow estimation network to obtain the reverse optical flow field image; Error and confidence level processing is performed on the forward optical flow field image and the reverse optical flow field image to obtain the optical flow filtered image.

3. The method according to claim 2, characterized in that, The step of performing error and confidence processing on the forward optical flow field image and the reverse optical flow field image to obtain the optical flow filtered image includes: Calculate the error between the pixels in the forward optical flow field image and the pixels in the reverse optical flow field image to obtain multiple error values; Confidence is calculated for each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence value corresponding to the pixel in the forward optical flow field image, wherein the optical flow amplitude is used to indicate the length of the optical flow vector; Each confidence value is compared with a preset confidence threshold. Pixels in the forward optical flow field image whose confidence value is greater than the confidence threshold are retained. Pixels in the forward optical flow field image whose confidence value is less than or equal to the confidence threshold are smoothed to obtain an optical flow filtered image.

4. The method according to claim 3, characterized in that, The calculation of the error between pixels in the forward optical flow field image and pixels in the reverse optical flow field image yields multiple error values, including: The error between pixels in the forward optical flow field image and pixels in the reverse optical flow field image is calculated using a preset error formula, resulting in multiple error values. The preset error formula is expressed as follows: E abs (p)=||F forward (p)+W(F forward ,F backward )(p)||2 p represents a pixel, F forward () is the forward optical flow vector at point p, F backward Let W() be the reverse optical flow vector, and E be a reprojection function. abs (p) represents the error value corresponding to the pixel. The error value corresponding to the pixel is obtained by calculating the Euclidean distance between the forward flow vector corresponding to the pixel and the distorted forward flow vector.

5. The method according to claim 3, characterized in that, The step of calculating the confidence level of each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence level value corresponding to the pixel in the forward optical flow field image includes: using a preset confidence level formula to calculate the confidence level of each error value and the preset optical flow amplitude corresponding to the error value to obtain the confidence level value corresponding to the pixel in the forward optical flow field image. The preset confidence level formula is expressed as follows: C(p) is the confidence value, p is a pixel in the forward optical flow field image, and σ abs and σ rel Both are adjustment parameters; the exp() function maps values ​​to the interval (0, 1], E abs (p) represents the error value corresponding to the pixel, E rel (p) represents the optical flow amplitude corresponding to the pixel.

6. The method according to claim 1, characterized in that, The pre-defined classification network includes a feature extraction module, a feature fusion module, and a classification module. The classification module includes a global average pooling layer, a feature expansion layer, an activation function layer, and a fully connected layer. The process of classifying the optical flow filtered image using a preset classification network to obtain classification results includes: The feature extraction layer is used to extract features from the optical flow filtered image to obtain multiple extracted features; The feature fusion module is used to fuse the extracted features to obtain fused features; The fused features are pooled using the global average pooling layer to obtain pooled features. The pooled features are expanded using the feature expansion layer to obtain one-dimensional feature data; The activation function layer is used to perform nonlinear processing on the one-dimensional feature data, and the fully connected layer is used to classify and map the nonlinearly processed data to output the classification result.

7. The method according to claim 1, characterized in that, Before acquiring the solder video data filtered by the camera lens, the method further includes: Determine whether the camera is working properly; if the camera is working properly, acquire the solder video data filtered by the camera lens. The camera has a bandpass filter in front of its lens, which is used to filter out the flame during welding.

8. A machine vision-based device for detecting the melting state of solder, characterized in that, The device includes: The data acquisition module is used to acquire the brazing filler video data filtered by the camera lens, and to sample the brazing filler video data to obtain two frames of the first brazing filler state image. The estimation calculation module is used to perform optical flow field estimation on two frames of the first solder state image using a preset optical flow estimation network to obtain a forward optical flow field image. The optical flow field estimation process is used to characterize the trend of the gray value of the pixel in the first solder state image. The filtering module is used to perform bidirectional consistency filtering on the forward optical flow field image to obtain an optical flow filtered image. The classification processing module is used to classify the optical flow filtered image using a preset classification network to obtain a classification result, which indicates whether the solder has melted. The adjustment module is used to sample the solder video data at a preset time interval when the classification result indicates that the solder has not melted, to obtain two frames of second solder state images, and to use the second solder state images as the first solder state images, and to perform the step of using a preset optical flow estimation network to estimate the optical flow field of the two frames of the first solder state images to obtain a forward optical flow field image.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, a communication bus, and a network interface. The processor, the memory, the user interface, and the network interface are respectively connected to the communication bus. The memory is used to store instructions. The user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images