Auxiliary material detection method and device based on 3D vision

Abstract

The invention relates to an auxiliary material detection method and device based on 3D vision. The method comprises the steps that an RGB image and a depth image of a to-be-detected workpiece are synchronously collected through a 3D vision sensor; inputting the RGB image and the depth image into a preset hybrid fusion semantic segmentation model to obtain a first segmentation result and a second fusion segmentation result of the auxiliary material; dynamically selecting the output of the hybrid fusion semantic segmentation model according to the data integrity of the RGB image and the depth image: when both the RGB image and the depth image are valid, taking a second fusion segmentation result as a final auxiliary segmentation result; when any one of the RGB image and the depth image is invalid, taking the first segmentation result as a final auxiliary material segmentation result; positioning an auxiliary material area according to a final auxiliary material segmentation result, and calculating position information and height information of the auxiliary material area relative to a workpiece reference; and comparing the position information and the height information with a preset qualified threshold value, and judging whether the auxiliary material has the defects of missing pasting, position deviation or repeated pasting or not.

Description

Technical Field

[0001] This application relates to the field of auxiliary material testing technology, and in particular to an auxiliary material testing method and apparatus based on 3D vision. Background Technology

[0002] In the precision manufacturing process of electronic products, such as smartphones, the bonding quality of various functional auxiliary materials (such as heat sinks, shielding films, cushioning foam, double-sided tape, etc.) directly determines the product's reliability, performance, and lifespan. Traditional manual visual inspection methods are inefficient, inconsistent in standards, and susceptible to fatigue, making them unsuitable for the demands of modern high-speed production lines. Therefore, automated auxiliary material inspection equipment based on machine vision has become a key element in improving production quality and efficiency.

[0003] Currently, the most widely used inspection method in the industry is based on two-dimensional (2D) vision. This method typically uses a high-resolution 2D industrial camera to acquire planar images of the workpiece to be inspected, and then analyzes the images using traditional image processing algorithms. Its core objective is to determine whether there are any missing parts of the auxiliary material and the deviation between its outline center and the preset theoretical position in a two-dimensional plane. This method transforms the inspection problem into the recognition of color and texture features, relying on sufficient color or grayscale contrast between the auxiliary material and the background.

[0004] Regarding the aforementioned technical methods, when the auxiliary material and the background color of the workpiece are similar, the distinguishability of the features will drop sharply, leading to inaccurate positioning or even missed detection. Secondly, 2D technology is completely unable to obtain the three-dimensional geometric information of the object, which makes it unable to detect the absolute or relative height of the auxiliary material, nor can it effectively deal with complex situations such as the auxiliary material surface being covered with a transparent protective film. Summary of the Invention

[0005] To address the issue that the distinguishability of features drops sharply when the background color of the auxiliary material is similar to that of the workpiece, leading to inaccurate positioning or even missed detection; and secondly, the inability of 2D technology to acquire the three-dimensional geometric information of the object, which makes it unable to detect the absolute or relative height of the auxiliary material, or to effectively handle complex situations such as the auxiliary material surface being covered with a transparent protective film, this application provides a method and apparatus for auxiliary material detection based on 3D vision.

[0006] Firstly, this application provides a 3D vision-based method for detecting auxiliary materials, employing the following technical solution: including: Simultaneously acquire RGB and depth images of the workpiece to be inspected using a 3D vision sensor; The RGB image and the depth image are input into a preset hybrid fusion semantic segmentation model to obtain a segmentation result image of the auxiliary material. The segmentation result image includes: a first segmentation result and a second fusion segmentation result. Based on the data integrity of the RGB image and the depth image, the output of the hybrid fusion semantic segmentation model is dynamically selected: when both the RGB image and the depth image are valid, the second fusion segmentation result is used as the final auxiliary segmentation result; when either the RGB image or the depth image is invalid, the first segmentation result is used as the final auxiliary segmentation result. Based on the final auxiliary material segmentation results, the auxiliary material area is located, and the position and height information of the auxiliary material area relative to the workpiece reference are calculated. The location information and height information are compared with a preset qualified threshold to determine whether there are defects such as missing application, positional offset or over-application of the auxiliary material; The construction and training of the hybrid fusion semantic segmentation model includes: constructing a dual-branch encoder to process the RGB image and the depth image respectively, and obtaining RGB features and depth features. The dual-branch encoder uses independent convolutional kernels in the first layer and weight-shared convolutional kernels in subsequent layers. A dynamic adaptive fusion module is constructed to simultaneously perform linear weighted fusion of the RGB features and the depth features to obtain a first fused feature; the RGB features and the depth features are respectively input into two independent auxiliary decoding heads to obtain a preliminary RGB segmentation result and a preliminary depth segmentation result; the preliminary RGB segmentation result and the preliminary depth segmentation result are fused by a dual attention fusion unit to obtain a second fused segmentation result; Construct a master decoder, input the first fused feature, and obtain the first segmentation result; The hybrid fusion semantic segmentation model is trained using a composite loss function, which includes at least: the cross-entropy loss between the first segmentation result and the real label, the cross-entropy loss between the second fusion segmentation result and the real label, and the feature distribution constraint loss between the RGB features and the deep features.

[0007] Preferably, before dynamically selecting the output of the hybrid fusion semantic segmentation model based on the data integrity of the RGB image and the depth image, the method further includes: Query the interface status of the 3D vision sensor. If the sensor reports an error or times out and does not return data, the corresponding image modality is determined to be invalid. Calculate the objective quality indicators of the RGB image and the depth image, and compare them with preset thresholds to determine whether the image quality of the RGB image and the depth image meets the standards; Calculate the feature differences between the RGB image and the depth image of the current frame and the historical frames. If the feature differences exceed a preset range, the data of the current frame is determined to be unreliable. Obtain the confidence maps of the preliminary RGB segmentation results and the preliminary depth segmentation results. If the average confidence is lower than a preset threshold, the data quality of the corresponding modality is determined to be unreliable. Based on the image modality, image quality, reliability of the current frame data, and data quality determination results, the data integrity of the RGB image and the depth image is determined to drive the path selection of the dynamic adaptive fusion module.

[0008] Preferably, constructing a dual-branch encoder, used to process the RGB image and the depth image respectively, to obtain RGB features and depth features, includes: The RGB image is input into the first independent convolutional layer of the RGB branch. The first independent convolutional layer of the RGB branch has a convolutional kernel adapted to three-channel color information to extract preliminary RGB features. The depth image is input into the first independent convolutional layer of the depth branch, which has a convolutional kernel adapted to single-channel geometric information to extract preliminary depth features. The initial RGB features and initial depth features are respectively input into a series of weight-shared convolutional layers. The weight-shared convolutional layers use the same convolution kernel parameters to perform convolution, non-linear activation and pooling operations on the features from the RGB branch and the depth branch respectively, forcing the two branches to learn from different input modalities and output the RGB features and the depth features that are consistent in abstract semantics. Meanwhile, when training the hybrid fusion semantic segmentation model, the feature distribution difference between the RGB features and the depth features at the same level output by the weight-shared convolutional layer is calculated; Based on the difference in feature distribution, the parameters of the dual-branch encoder are optimized using a feature distribution constraint loss function to minimize the distance between the RGB features and the depth features in the shared semantic space.

[0009] Preferably, the step of constructing a dynamic adaptive fusion module simultaneously performs linear weighted fusion of the RGB features and the depth features to obtain a first fused feature; inputting the RGB features and the depth features into two independent auxiliary decoding heads respectively to obtain a preliminary RGB segmentation result and a preliminary depth segmentation result; fusing the preliminary RGB segmentation result and the preliminary depth segmentation result through a dual attention fusion unit to obtain a second fused segmentation result, including: Assign a learnable first weight coefficient to the RGB feature and another learnable second weight coefficient to the depth feature, the sum of the first weight coefficient and the second weight coefficient being 1; add the weighted RGB feature and the weighted depth feature element by element to obtain the first fused feature; The RGB features and the depth features are respectively input into two independent auxiliary decoding heads, and upsampling and pixel-level classification are performed respectively to obtain the preliminary RGB segmentation result and the preliminary depth segmentation result. The preliminary RGB segmentation result and the preliminary depth segmentation result are both feature maps containing the confidence of each pixel in different categories. The second fused segmentation result is generated by fusing the RGB preliminary segmentation result and the depth preliminary segmentation result through a dual attention fusion unit.

[0010] Preferably, the step of fusing the preliminary RGB segmentation result and the preliminary depth segmentation result through a dual attention fusion unit to generate the second fused segmentation result includes: Global average pooling is calculated for each semantic category channel of the RGB preliminary segmentation result and the depth preliminary segmentation result respectively to generate a channel attention vector representing the overall confidence of each category; the channel attention vector is processed by a fully connected layer and a non-linear activation function to obtain the channel weights used to adjust the importance of each semantic category; The preliminary RGB segmentation results and the preliminary depth segmentation results are concatenated along the channel dimension, and a spatial attention map is generated through a convolution operation. The spatial attention map is used to evaluate the reliability of different positions from the RGB mode and the depth mode in the spatial dimension. The channel weights are used to recalibrate the preliminary RGB segmentation result and the preliminary depth segmentation result in terms of channel dimensions. The spatial attention map is then used to perform spatial dimension weighted fusion of the two calibrated results. Finally, the results are normalized using the Softmax function to generate the second fused segmentation result.

[0011] Preferably, constructing a master decoder, inputting the first fused feature, and obtaining the first segmentation result includes: The main decoder consists of multiple cascaded decoding modules. Each decoding module sequentially upsamples the input feature map. The input of the first decoding module is the first fused feature, and the output is the feature map. The output of the previous decoding module is the input of the next decoding module, thereby increasing the spatial resolution of the feature map to the target scale. The upsampled feature map is then fused with RGB features and / or depth features from the corresponding resolution level in the dual-branch encoder. The fused feature maps are subjected to convolution and nonlinear sum operations to refine the feature representation; The last decoding module of the main decoder outputs a feature map with the same resolution as the original input image. Then, a 1x1 convolutional layer is used as a classifier to calculate the confidence score of each pixel in the feature map belonging to each semantic category. Finally, the category of each pixel is determined by the Argmax operation to generate the first segmentation result.

[0012] Preferably, after performing convolution and nonlinear sum operations on the fused feature maps to refine the feature representation, the method further includes: After the output of at least one intermediate decoding module of the main decoder, an auxiliary segmentation head is added to downsample the feature map output by the intermediate decoding module to the corresponding scale and calculate the cross-entropy loss with the downsampled real label to obtain the intermediate supervision loss. The intermediate supervision loss is weighted and summed with the cross-entropy loss calculated by the first segmentation result and the real label at the final output of the main decoder to obtain the fused cross-entropy loss of the first segmentation result and the real label.

[0013] Preferably, the step of locating the auxiliary material area based on the final auxiliary material segmentation result and calculating the position and height information of the auxiliary material area relative to the workpiece reference includes: Connectivity analysis was performed on the final excipient segmentation results to extract all excipient regions identified as excipients. According to the predefined reference hole positioning template, the reference hole of the workpiece is positioned in the RGB image or the depth image, and the XY pixel coordinate system and the Z-axis depth coordinate system are established with the center of the reference hole of the workpiece as the origin of the spatial coordinate system. Calculate the geometric center point of the auxiliary material area to obtain its center coordinates in the XY pixel coordinate system; Calculate the pixel distance difference between the center coordinates and the coordinates of the reference hole center in the X and Y directions; Based on the camera's intrinsic calibration data, the pixel distance difference is converted into a position offset in actual physical space to obtain the position information; The geometric center point of the auxiliary material area is mapped onto the depth image to obtain the depth value of the geometric center point, which is used as the height measurement value of the auxiliary material surface. Select at least three non-collinear points on the workpiece reference plane, obtain their depth values ​​from the depth image, and fit a reference plane using the three non-collinear points. The distance from the measured height of the auxiliary material surface to the fitting reference plane along the normal direction is calculated to obtain the height information. Based on the height information, it is determined whether the auxiliary material has a height deviation or a weight-adhesion defect.

[0014] Preferably, the method further includes: The defect determination results, the corresponding RGB images and depth images, and the final auxiliary material segmentation results are stored in an online database; When the operator confirms or corrects the automatic judgment result in the system, a training sample containing the corrected label is generated. The hybrid semantic segmentation model is incrementally trained periodically using the collected training samples to optimize the model parameters.

[0015] Secondly, this application discloses a 3D vision-based auxiliary material inspection device, which adopts the following technical solution, including: The vision inspection module is used to simultaneously acquire RGB and depth images of the workpiece to be inspected using a 3D vision sensor; The semantic segmentation module is used to input the RGB image and the depth image into a preset hybrid fusion semantic segmentation model to obtain a segmentation result image of the auxiliary material. The segmentation result image includes: a first segmentation result and a second fusion segmentation result. The auxiliary segmentation module is used to dynamically select the output of the hybrid fusion semantic segmentation model based on the data integrity of the RGB image and the depth image: when both the RGB image and the depth image are valid, the second fusion segmentation result is used as the final auxiliary segmentation result; when either the RGB image or the depth image is invalid, the first segmentation result is used as the final auxiliary segmentation result. The auxiliary material positioning module is used to locate the auxiliary material area based on the final auxiliary material segmentation result, and to calculate the position and height information of the auxiliary material area relative to the workpiece reference. The auxiliary material detection module is used to compare the position information and the height information with a preset qualified threshold to determine whether there are defects such as missing application, positional offset or over-application of the auxiliary material. The semantic segmentation module is also used to construct a dual-branch encoder, which is used to process the RGB image and the depth image respectively to obtain RGB features and depth features. The dual-branch encoder uses independent convolution kernels in the first layer and weight-shared convolution kernels in subsequent layers. A dynamic adaptive fusion module is constructed to simultaneously perform linear weighted fusion of the RGB features and the depth features to obtain a first fused feature; the RGB features and the depth features are respectively input into two independent auxiliary decoding heads to obtain a preliminary RGB segmentation result and a preliminary depth segmentation result; the preliminary RGB segmentation result and the preliminary depth segmentation result are fused by a dual attention fusion unit to obtain a second fused segmentation result; Construct a master decoder, input the first fused feature, and obtain the first segmentation result; The hybrid fusion semantic segmentation model is trained using a composite loss function, which includes at least: the cross-entropy loss between the first segmentation result and the real label, the cross-entropy loss between the second fusion segmentation result and the real label, and the feature distribution constraint loss between the RGB features and the deep features.

[0016] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the 3D vision-based auxiliary material detection method described above.

[0017] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above for the 3D vision-based auxiliary material detection method.

[0018] In summary, this application synchronously acquires RGB and depth images of the workpiece using a 3D sensor, which are then input into an innovative hybrid fusion model. This model employs a dual-branch encoder structure, where the first layer independently processes data from different modalities, while subsequent layers share weights to force the network to learn unified semantic features, and this characteristic is reinforced through feature distribution constraint loss. The core of the model is a dynamic adaptive fusion module that executes two fusion strategies in parallel: one is to linearly weight the features to generate a first fused feature, which is then output as the first segmentation result by the main decoder; the other is to generate a more accurate second fused segmentation result through independent auxiliary decoders and a dual attention fusion unit. During inference, the system evaluates the data integrity of the RGB and depth images in real time and dynamically selects the optimal segmentation result as the output, thus providing reliable detection even if any sensor fails, achieving high robustness. Finally, the system locates the auxiliary material area based on the segmentation result and accurately determines whether there are defects such as missing attachment, positional misalignment, or over-attachment by calculating its XY-direction offset relative to the workpiece's reference hole and its Z-axis height relative to the fitted reference plane. By leveraging information complementarity and adaptive decision-making mechanisms, this technology overcomes the limitations of traditional 2D inspection, which is greatly affected by color and lighting and cannot measure height information, thus achieving high-precision inspection of the quality of auxiliary material bonding in complex industrial environments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a 3D vision-based auxiliary material detection method.

[0020] Figure 2 This is a flowchart illustrating the construction process of the hybrid fusion semantic segmentation model in this application.

[0021] Figure 3 This is a system display diagram of the detection of the location and height of auxiliary materials in this application.

[0022] Figure 4 This is a structural block diagram of an auxiliary material detection device based on 3D vision. Detailed Implementation

[0023] In some embodiments, to confirm the data integrity of RGB and depth images, the corresponding processing steps are as follows: Query the interface status of the 3D vision sensor; if the sensor reports an error or fails to return data after a timeout, the corresponding image modality is determined to be invalid; calculate the objective quality indicators of the RGB and depth images and compare them with preset thresholds to determine whether the image quality of the RGB and depth images meets the standards; calculate the feature differences between the RGB and depth images of the current frame and historical frames; if the feature differences exceed a preset range, the current frame data is determined to be unreliable; obtain the confidence maps of the preliminary RGB segmentation results and the preliminary depth segmentation results; if the average confidence is lower than a preset threshold, the data quality of the corresponding modality is determined to be unreliable; based on the image modality, image quality, reliability of the current frame data, and the data quality determination results, determine the data integrity of the RGB and depth images to drive the path selection of the dynamic adaptive fusion module.

[0024] Specifically, the process begins by querying the 3D sensor interface status to directly determine the modal validity at the hardware level. Then, objective metrics of the RGB and depth images are analyzed from an image quality perspective to rule out data failures caused by imaging anomalies. Simultaneously, by calculating the feature differences between the current frame and historical frames, abnormal situations such as data abrupt changes are identified from a temporal continuity perspective. Finally, combined with the model's internal confidence information, the average confidence of the preliminary segmentation results from the RGB and depth branches is analyzed to verify data reliability. This comprehensive judgment, integrating hardware status, image quality, temporal consistency, and model introspection, provides a decision-making basis for the path selection of the dynamic adaptive fusion module.

[0025] In some embodiments, refer to Figure 2 The specific steps for constructing a hybrid semantic segmentation model are as follows: 1. Construction of a dual-branch encoder The RGB image is input into the first independent convolutional layer of the RGB branch, which has a convolutional kernel adapted to three-channel color information to extract preliminary RGB features. The depth image is input into the first independent convolutional layer of the depth branch, which has a convolutional kernel adapted to single-channel geometric information to extract preliminary depth features. The preliminary RGB features and preliminary depth features are then input into a series of weight-shared convolutional layers. The weight-shared convolutional layers use the same convolutional kernel parameters to perform convolution, non-linear activation, and pooling operations on the features from the RGB and depth branches, respectively, forcing the two branches to learn from different input modalities and output RGB and depth features that are consistent in abstract semantics. At the same time, when training the hybrid fusion semantic segmentation model, the feature distribution difference between the RGB features and depth features at the same level output by the weight-shared convolutional layers is calculated. Based on the feature distribution difference, the parameters of the dual-branch encoder are optimized using a feature distribution constraint loss function to minimize the distance between RGB features and depth features in the shared semantic space.

[0026] Let the feature map output by the RGB branch be... The feature map output by the deep branch is H, W, and C represent the height, width, and number of channels of the feature map, respectively. The feature distribution constraint loss function is shown in the following formula: ; in, Let represent the feature values ​​of the RGB feature map and the depth feature map at position (h, w, c), respectively.

[0027] Specifically, the model first extracts texture features from the RGB image and geometric features from the depth image through modality-specific independent convolutional layers. Then, the acquired preliminary features are fed into weight-shared convolutional layers for collaborative processing, forcing the two modalities to form a unified representation in the high-level semantic space. At the same time, during model training, the network parameters are continuously optimized by calculating and minimizing the distribution difference between the output features of the corresponding layers of the two branches and using feature distribution constraint loss. This effectively eliminates the domain differences between modalities and provides a highly consistent semantic feature foundation for the subsequent dynamic adaptive fusion module.

[0028] 2. Construction of the Dynamic Adaptive Fusion Module A learnable first weight coefficient is assigned to the RGB features, and another learnable second weight coefficient is assigned to the depth features. The sum of the first and second weight coefficients is 1. The weighted RGB features and the weighted depth features are added element-wise to obtain the first fused feature. The RGB features and depth features are input into two independent auxiliary decoding heads, respectively, and upsampling and pixel-level classification are performed to obtain the preliminary RGB segmentation result and the preliminary depth segmentation result. Both the preliminary RGB segmentation result and the preliminary depth segmentation result are feature maps containing the confidence of each pixel in different categories. The preliminary RGB segmentation result and the preliminary depth segmentation result are fused through a dual attention fusion unit to generate the second fused segmentation result.

[0029] A second fused segmentation result is generated by fusing the initial RGB segmentation result and the initial depth segmentation result through a dual attention fusion unit, including: Global average pooling is calculated for each semantic category channel of the preliminary RGB and depth segmentation results to generate a channel attention vector representing the overall confidence of each category. The channel attention vector is processed by a fully connected layer and a non-linear activation function to obtain channel weights used to adjust the importance of each semantic category. The preliminary RGB and depth segmentation results are concatenated along the channel dimension, and a spatial attention map is generated through convolution. The spatial attention map is used to evaluate the reliability of different positions from the RGB and depth modal in the spatial dimension. The channel weights are used to recalibrate the preliminary RGB and depth segmentation results along the channel dimension, and the spatial attention map is used to perform a weighted fusion of the two calibrated results along the spatial dimension. Finally, the results are normalized using the Softmax function to generate a second fused segmentation result.

[0030] First fusion feature The calculation formula is: ; in, and These are learnable scalar coefficients.

[0031] Let the initial RGB segmentation result be... The initial depth segmentation result is N is the number of semantic categories, and the channel attention weights are calculated as follows: ; in This is the channel description vector obtained through global average pooling. and For the weights of the fully connected layer, It is the ReLU activation function. Use the Sigmoid activation function; The calculation process for the spatial attention map is as follows: ; The final second fusion segmentation result is as follows: ; in, This indicates element-wise multiplication, and 1 represents a matrix consisting entirely of 1s.

[0032] Specifically, in the feature fusion path, RGB features and depth features are adaptively weighted and summed using learnable weight coefficients to generate a first fusion feature with modal complementarity. In the decision fusion path, RGB features and depth features are processed by independent decoding heads to generate preliminary segmentation results, which are then integrated by a dual attention fusion unit. This unit first analyzes the confidence differences of the two modalities across semantic categories using a channel attention mechanism to generate adaptive weights for different categories. Subsequently, it evaluates the reliability distribution of the two modalities at different spatial locations using a spatial attention mechanism. Finally, it performs collaborative calibration and weighted fusion of the preliminary segmentation results in both channel and spatial dimensions, ultimately outputting a second fusion segmentation result with higher accuracy and boundary consistency. This dual-path design ensures the system's baseline performance while improving detection accuracy under optimal conditions through a refined attention fusion mechanism.

[0033] 3. Construction of the main decoder The main decoder consists of multiple cascaded decoding modules. Each decoding module sequentially upsamples the input feature map. The input of the first decoding module is the first fused feature, and the output is the feature map. The output of the previous decoding module is the input of the next decoding module, thus increasing the spatial resolution of the feature map to the target scale. The upsampled feature map is then fused with RGB features and / or depth features from the corresponding resolution level in the dual-branch encoder. Specifically, the RGB features and depth features are first channel-adjusted through a 1x1 convolutional layer, then the channel weights are recalibrated through a compression excitation module, and finally the processed dual-branch features are element-wise added to the upsampled feature map. The fused feature map is then subjected to convolution and non-linear sum operations to refine the feature representation. The last decoding module of the main decoder outputs a feature map with the same resolution as the original input image. Subsequently, a 1x1 convolutional layer is used as a classifier to calculate the confidence score of each pixel in the feature map for each semantic category. Finally, the Argmax operation is used to determine the category of each pixel, generating the first segmentation result.

[0034] Furthermore, refer to Figure 2The hybrid fusion semantic segmentation model is trained using a composite loss function, including: adding an auxiliary segmentation head after the output of at least one intermediate decoding module of the main decoder, downsampling the feature map output by the intermediate decoding module to the corresponding scale, and calculating the cross-entropy loss with the downsampled real label to obtain the intermediate supervision loss. The intermediate supervision loss is then weighted and summed with the cross-entropy loss calculated from the first segmentation result and the real label at the end of the main decoder output to obtain the fusion cross-entropy loss of the first segmentation result and the real label.

[0035] The fusion cross-entropy loss is constructed as follows: Let the first segmentation result output by the main decoder be... The true label is Y, and the feature map output by the intermediate decoding module is... The true label after downsampling is Where k=1,2,…,K represents the intermediate supervision level, and the formula for the fusion cross-entropy loss function is as follows: ; Master decoder loss: ; Intermediate monitoring loss: ; Where, here λ and These are the weighting coefficients for each loss term. and Let be the height and width of the k-th intermediate layer feature map.

[0036] Specifically, the main decoder adopts a cascaded structure, gradually restoring the spatial resolution of the feature map through multiple cascaded decoding modules. The first decoding module takes the first fused feature as input, and each module sequentially performs three core operations: upsampling, feature fusion, and feature refinement. In the feature fusion stage, after unifying the channel dimension through 1×1 convolution, the multi-level feature weights from the dual-branch encoder are adaptively recalibrated using a compression excitation module. Finally, the deep and shallow features are effectively fused by element-wise addition. In the training stage, a multi-scale supervision mechanism is introduced. An auxiliary segmentation head is added to the intermediate decoding layer to calculate the intermediate supervision loss, which together with the cross-entropy loss of the final output layer constitutes the fused cross-entropy loss. This hierarchical supervision strategy effectively ensures gradient propagation and detail recovery capabilities during feature reconstruction. Finally, the first segmentation result is generated through a classifier and Argmax operation.

[0037] In some embodiments, step S40 specifically includes the following steps: performing connected component analysis on the final auxiliary material segmentation result to extract all auxiliary material regions identified as auxiliary materials; locating the reference hole of the workpiece in the RGB image or depth image according to the predefined reference hole positioning template, and establishing an XY pixel coordinate system and a Z-axis depth coordinate system with the center of the reference hole of the workpiece as the origin of the spatial coordinate system; calculating the geometric center point of the auxiliary material region to obtain its center coordinates in the XY pixel coordinate system; calculating the pixel distance difference between the center coordinates and the coordinates of the reference hole center in the X and Y directions; according to The camera's intrinsic calibration data converts pixel distance differences into positional offsets in actual physical space to obtain positional information. The geometric center point of the auxiliary material area is mapped onto the depth image to obtain the depth value of the geometric center point, which is used as the height measurement value of the auxiliary material surface. At least three non-collinear points are selected on the workpiece reference plane, and their depth values ​​are obtained from the depth image. A fitting reference plane is fitted using the three non-collinear points. The distance from the height measurement value of the auxiliary material surface to the fitting reference plane along the normal direction is calculated to obtain height information. Based on the height information, it is determined whether the auxiliary material has excessive height or excessive weight.

[0038] Reference Figure 3 Specifically, firstly, connected component analysis is performed on the segmentation results to locate all auxiliary material areas. Simultaneously, a spatial coordinate system with the center of the hole as the origin is established based on the reference hole positioning template. By calculating the pixel coordinate difference between the geometric center of the auxiliary material area and the center of the reference hole, and combining it with the camera intrinsic parameters, it is converted into the position offset in the actual physical space, thereby obtaining accurate XY direction position information. In terms of height measurement, the center point of the auxiliary material is mapped to the depth image to obtain the surface height value. Then, by selecting at least three non-collinear points on the reference plane, a reference plane is fitted. Finally, the distance from the auxiliary material surface to the plane along the normal direction is calculated as the height information. This method effectively eliminates the measurement error caused by the tilt of the workpiece placement, can accurately identify height deviation and over-weighting defects, and realizes a complete mapping from two-dimensional pixel coordinates to three-dimensional physical space.

[0039] In some embodiments, considering the issue of optimizing model parameters, the corresponding processing steps are as follows: the defect judgment result, the corresponding RGB image and depth image, and the final auxiliary material segmentation result are stored in an online database; when the operator confirms or corrects the automatic judgment result in the system, a training sample containing the corrected label is generated; the collected training samples are used periodically to incrementally train the hybrid fusion semantic segmentation model to optimize the model parameters.

[0040] The implementation principle of the auxiliary material detection method based on 3D vision in this application is as follows: In this application, RGB and depth images of the workpiece are simultaneously acquired by a 3D sensor and then input into an innovative hybrid fusion model. This model adopts a dual-branch encoder structure. Its first layer independently processes data from different modalities, while subsequent layers share weights to force the network to learn unified semantic features, and this characteristic is strengthened through feature distribution constraint loss. The core of the model is a dynamic adaptive fusion module that executes two fusion strategies in parallel: one is to perform linear weighted fusion of features to generate a first fusion feature, which is then output as a first segmentation result by the main decoder; the other is to generate a more accurate second fusion segmentation result through independent auxiliary decoder heads and dual attention fusion units. During inference, the system evaluates the data integrity of the RGB and depth images in real time and dynamically selects the optimal segmentation result as the output, thus providing reliable detection even if any sensor fails, achieving high robustness. Finally, the system locates the auxiliary material area based on the segmentation result and accurately determines whether there are defects such as missing attachment, positional offset, or over-attachment by calculating its XY direction positional offset relative to the workpiece reference hole and its Z-axis height relative to the fitted reference plane. By leveraging information complementarity and adaptive decision-making mechanisms, this technology overcomes the limitations of traditional 2D inspection, which is greatly affected by color and lighting and cannot measure height information, thus achieving high-precision inspection of the quality of auxiliary material bonding in complex industrial environments.

[0041] Figure 1 This is a flowchart illustrating a 3D vision-based auxiliary material detection method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0042] Based on the same technical concept, referring to Figure 4 This application also provides a 3D vision-based auxiliary material detection device, which adopts the following technical solution: the device includes: The vision inspection module is used to simultaneously acquire RGB and depth images of the workpiece to be inspected using a 3D vision sensor; The semantic segmentation module is used to input RGB images and depth images into a preset hybrid fusion semantic segmentation model to obtain the segmentation result image of the auxiliary material. The segmentation result image includes: a first segmentation result and a second fusion segmentation result. The auxiliary segmentation module is used to dynamically select the output of the hybrid fusion semantic segmentation model based on the data integrity of the RGB image and the depth image: when both the RGB image and the depth image are valid, the second fusion segmentation result is used as the final auxiliary segmentation result; when either the RGB image or the depth image is invalid, the first segmentation result is used as the final auxiliary segmentation result. The auxiliary material positioning module is used to locate the auxiliary material area based on the final auxiliary material segmentation result, and to calculate the position and height information of the auxiliary material area relative to the workpiece reference. The auxiliary material detection module is used to compare the position and height information with the preset qualified threshold to determine whether there are defects such as missing, positional offset or over-attachment of auxiliary materials. The semantic segmentation module is also used to build a dual-branch encoder, which processes RGB images and depth images respectively, and obtains RGB features and depth features. The dual-branch encoder uses independent convolutional kernels in the first layer and weight-shared convolutional kernels in subsequent layers. A dynamic adaptive fusion module is constructed to perform linear weighted fusion of RGB features and depth features to obtain the first fused feature. The RGB features and depth features are input into two independent auxiliary decoding heads to obtain the preliminary RGB segmentation result and the preliminary depth segmentation result. The preliminary RGB segmentation result and the preliminary depth segmentation result are fused by a dual attention fusion unit to obtain the second fused segmentation result. Construct a master decoder, input the first fused feature, and obtain the first segmentation result; The hybrid fusion semantic segmentation model is trained using a composite loss function, which includes at least: the cross-entropy loss between the first segmentation result and the real label, the cross-entropy loss between the second fusion segmentation result and the real label, and the feature distribution constraint loss between RGB features and deep features.

[0043] In some embodiments, the vision detection module is also used to query the interface status of the 3D vision sensor. If the sensor reports an error or fails to return data after a timeout, the corresponding image modality is determined to be invalid. Calculate the objective quality indicators of RGB and depth images and compare them with preset thresholds to determine whether the image quality of RGB and depth images meets the standards. Calculate the feature differences between the RGB and depth images of the current frame and the historical frames. If the feature differences exceed the preset range, the data of the current frame is determined to be unreliable. Obtain confidence maps of the preliminary RGB segmentation results and the preliminary depth segmentation results. If the average confidence score is lower than a preset threshold, the data quality of the corresponding modality is determined to be unreliable. Based on image modality, image quality, reliability of current frame data, and data quality determination results, the data integrity of RGB and depth images is judged to drive the path selection of the dynamic adaptive fusion module.

[0044] In some embodiments, the auxiliary material segmentation module is specifically used to input the RGB image into the first independent convolutional layer of the RGB branch. The first independent convolutional layer of the RGB branch has a convolutional kernel adapted to three-channel color information to extract preliminary RGB features. The depth image is input into the first independent convolutional layer of the depth branch, which has a convolutional kernel adapted to single-channel geometric information to extract preliminary depth features. The initial RGB features and initial depth features are respectively input into a series of weight-shared convolutional layers. The weight-shared convolutional layers use the same convolutional kernel parameters to perform convolution, non-linear activation and pooling operations on the features from the RGB branch and the depth branch respectively, forcing the two branches to learn from different input modalities and output RGB features and depth features that are consistent in abstract semantics. Meanwhile, when training the hybrid fusion semantic segmentation model, the feature distribution difference between RGB features and deep features at the same level output by the weight-shared convolutional layer is calculated; Based on the difference in feature distribution, a feature distribution-constrained loss function is used to optimize the parameters of the dual-branch encoder, minimizing the distance between RGB features and deep features in the shared semantic space.

[0045] In some embodiments, the semantic segmentation module is specifically used to assign a learnable first weight coefficient to the RGB features and another learnable second weight coefficient to the deep features, wherein the sum of the first weight coefficient and the second weight coefficient is 1; and to add the weighted RGB features and the weighted deep features element by element to obtain the first fused feature. The RGB features and depth features are input into two independent auxiliary decoding heads, and upsampling and pixel-level classification are performed respectively to obtain the preliminary RGB segmentation result and the preliminary depth segmentation result. Both the preliminary RGB segmentation result and the preliminary depth segmentation result are feature maps containing the confidence of each pixel in different categories. A second fused segmentation result is generated by fusing the initial RGB segmentation result and the initial depth segmentation result through a dual attention fusion unit.

[0046] In some embodiments, the semantic segmentation module is specifically used to calculate global average pooling for each semantic category channel of the RGB preliminary segmentation result and the depth preliminary segmentation result respectively, to generate a channel attention vector representing the overall confidence of each category; the channel attention vector is processed by a fully connected layer and a non-linear activation function to obtain the channel weights used to adjust the importance of each semantic category; The preliminary RGB segmentation results and the preliminary depth segmentation results are concatenated along the channel dimension, and a spatial attention map is generated through a convolution operation. The spatial attention map is used to evaluate the reliability of different positions from the RGB mode and the depth mode in the spatial dimension. The channel dimensions of the preliminary RGB segmentation results and the preliminary depth segmentation results are recalibrated using channel weights. The spatial attention map is then used to perform spatial dimension weighted fusion of the two calibrated results. Finally, the results are normalized using the Softmax function to generate the second fused segmentation result.

[0047] In some embodiments, the semantic segmentation module is specifically used in a main decoder composed of multiple cascaded decoding modules. Each decoding module sequentially upsamples the input feature map. The input of the first decoding module is the first fused feature, and the output is the feature map. The output of the previous decoding module is the input of the next decoding module, thereby increasing the spatial resolution of the feature map to the target scale. The upsampled feature map is then fused with RGB features and / or depth features from the corresponding resolution level in the dual-branch encoder. The fused feature maps are subjected to convolution and nonlinear sum operations to refine the feature representation; The last decoding module of the main decoder outputs a feature map with the same resolution as the original input image. Then, a 1x1 convolutional layer is used as a classifier to calculate the confidence of each pixel in the feature map belonging to each semantic category. Finally, the Argmax operation is used to determine the category of each pixel, generating the first segmentation result.

[0048] In some embodiments, the semantic segmentation module is specifically used to add an auxiliary segmentation head after the output of at least one intermediate decoding module of the main decoder, downsample the feature map output by the intermediate decoding module to the corresponding scale, and calculate the cross-entropy loss with the downsampled real label to obtain the intermediate supervision loss. The intermediate supervision loss is weighted and summed with the cross-entropy loss calculated by the first segmentation result finally output by the main decoder and the real label to obtain the fused cross-entropy loss of the first segmentation result and the real label.

[0049] In some embodiments, the auxiliary material positioning module is specifically used to perform connected component analysis on the final auxiliary material segmentation result and extract all auxiliary material regions identified as auxiliary materials. Based on the predefined reference hole positioning template, locate the reference hole of the workpiece in the RGB image or depth image, and establish the XY pixel coordinate system and the Z-axis depth coordinate system with the center of the reference hole of the workpiece as the origin of the spatial coordinate system. Calculate the geometric center point of the auxiliary material area to obtain its center coordinates in the XY pixel coordinate system; Calculate the pixel distance difference between the center coordinates and the coordinates of the reference hole center in the X and Y directions; Based on the camera's intrinsic calibration data, the pixel distance difference is converted into a position offset in the actual physical space to obtain position information; Map the geometric center point of the auxiliary material area onto the depth image to obtain the depth value of the geometric center point, which is used as the height measurement value of the auxiliary material surface. Select at least three non-collinear points on the workpiece reference plane, obtain their depth values ​​from the depth image, and use the three non-collinear points to fit a fitting reference plane. The distance from the measured height of the auxiliary material surface to the fitting reference plane along the normal direction is calculated to obtain the height information. Based on the height information, it is determined whether the auxiliary material has a height deviation or a weight-adhesion defect.

[0050] In some embodiments, the auxiliary material positioning module is also used to store the defect determination result, the corresponding RGB image and depth image, and the final auxiliary material segmentation result into an online database; When the operator confirms or corrects the automatic judgment result in the system, a training sample containing the corrected label is generated. The hybrid semantic segmentation model is incrementally trained periodically using the collected training samples to optimize the model parameters.

[0051] This application also discloses a control device.

[0052] Specifically, the control device includes a memory and a processor, with the memory storing a computer program that can be loaded by the processor and executed to perform the aforementioned 3D vision-based auxiliary material detection method.

[0053] This application also discloses a computer-readable storage medium.

[0054] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the 3D vision-based auxiliary material detection method described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0055] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for detecting auxiliary materials based on 3D vision, characterized in that, include: Simultaneously acquire RGB and depth images of the workpiece to be inspected using a 3D vision sensor; The RGB image and the depth image are input into a preset hybrid fusion semantic segmentation model to obtain a segmentation result image of the auxiliary material. The segmentation result image includes: a first segmentation result and a second fusion segmentation result. Based on the data integrity of the RGB image and the depth image, the output of the hybrid fusion semantic segmentation model is dynamically selected: when both the RGB image and the depth image are valid, the second fusion segmentation result is used as the final auxiliary segmentation result; when either the RGB image or the depth image is invalid, the first segmentation result is used as the final auxiliary segmentation result. Based on the final auxiliary material segmentation results, the auxiliary material area is located, and the position and height information of the auxiliary material area relative to the workpiece reference are calculated. The location information and height information are compared with a preset qualified threshold to determine whether there are defects such as missing application, positional offset or over-application of the auxiliary material; The construction and training of the hybrid fusion semantic segmentation model includes: constructing a dual-branch encoder to process the RGB image and the depth image respectively, and obtaining RGB features and depth features. The dual-branch encoder uses independent convolutional kernels in the first layer and weight-shared convolutional kernels in subsequent layers. A dynamic adaptive fusion module is constructed to simultaneously perform linear weighted fusion of the RGB features and the depth features to obtain a first fused feature; the RGB features and the depth features are respectively input into two independent auxiliary decoding heads to obtain a preliminary RGB segmentation result and a preliminary depth segmentation result; the preliminary RGB segmentation result and the preliminary depth segmentation result are fused by a dual attention fusion unit to obtain a second fused segmentation result; Construct a master decoder, input the first fused feature, and obtain the first segmentation result; The hybrid fusion semantic segmentation model is trained using a composite loss function, which includes at least: the cross-entropy loss between the first segmentation result and the real label, the cross-entropy loss between the second fusion segmentation result and the real label, and the feature distribution constraint loss between the RGB features and the deep features.

2. The auxiliary material detection method based on 3D vision according to claim 1, characterized in that, Before dynamically selecting the output of the hybrid fusion semantic segmentation model based on the data integrity of the RGB image and the depth image, the method further includes: Query the interface status of the 3D vision sensor. If the sensor reports an error or times out and does not return data, the corresponding image modality is determined to be invalid. Calculate the objective quality indicators of the RGB image and the depth image, and compare them with preset thresholds to determine whether the image quality of the RGB image and the depth image meets the standards; Calculate the feature differences between the RGB image and the depth image of the current frame and the historical frames. If the feature differences exceed a preset range, the data of the current frame is determined to be unreliable. Obtain the confidence maps of the preliminary RGB segmentation results and the preliminary depth segmentation results. If the average confidence is lower than a preset threshold, the data quality of the corresponding modality is determined to be unreliable. Based on the image modality, image quality, reliability of the current frame data, and data quality determination results, the data integrity of the RGB image and the depth image is determined to drive the path selection of the dynamic adaptive fusion module.

3. The auxiliary material detection method based on 3D vision according to claim 1, characterized in that, The construction of a dual-branch encoder, used to process the RGB image and the depth image respectively, to obtain RGB features and depth features, includes: The RGB image is input into the first independent convolutional layer of the RGB branch. The first independent convolutional layer of the RGB branch has a convolutional kernel adapted to three-channel color information to extract preliminary RGB features. The depth image is input into the first independent convolutional layer of the depth branch, which has a convolutional kernel adapted to single-channel geometric information to extract preliminary depth features. The initial RGB features and initial depth features are respectively input into a series of weight-shared convolutional layers. The weight-shared convolutional layers use the same convolution kernel parameters to perform convolution, non-linear activation and pooling operations on the features from the RGB branch and the depth branch respectively, forcing the two branches to learn from different input modalities and output the RGB features and the depth features that are consistent in abstract semantics. Meanwhile, when training the hybrid fusion semantic segmentation model, the feature distribution difference between the RGB features and the depth features at the same level output by the weight-shared convolutional layer is calculated; Based on the difference in feature distribution, the parameters of the dual-branch encoder are optimized using a feature distribution constraint loss function to minimize the distance between the RGB features and the depth features in the shared semantic space.

4. The auxiliary material detection method based on 3D vision according to claim 1, characterized in that, A dynamic adaptive fusion module is constructed to simultaneously perform linear weighted fusion of the RGB features and the depth features to obtain a first fused feature; the RGB features and the depth features are respectively input into two independent auxiliary decoding heads to obtain preliminary RGB segmentation results and preliminary depth segmentation results; the preliminary RGB segmentation results and preliminary depth segmentation results are fused through a dual attention fusion unit to obtain a second fused segmentation result, including: Assign a learnable first weight coefficient to the RGB feature and another learnable second weight coefficient to the depth feature, the sum of the first weight coefficient and the second weight coefficient being 1; add the weighted RGB feature and the weighted depth feature element by element to obtain the first fused feature; The RGB features and the depth features are respectively input into two independent auxiliary decoding heads, and upsampling and pixel-level classification are performed respectively to obtain the preliminary RGB segmentation result and the preliminary depth segmentation result. The preliminary RGB segmentation result and the preliminary depth segmentation result are both feature maps containing the confidence of each pixel in different categories. The second fused segmentation result is generated by fusing the RGB preliminary segmentation result and the depth preliminary segmentation result through a dual attention fusion unit.

5. The auxiliary material detection method based on 3D vision according to claim 4, characterized in that, The step of fusing the preliminary RGB segmentation result and the preliminary depth segmentation result through a dual attention fusion unit to generate the second fused segmentation result includes: Global average pooling is calculated for each semantic category channel of the RGB preliminary segmentation result and the depth preliminary segmentation result respectively to generate a channel attention vector representing the overall confidence of each category; the channel attention vector is processed by a fully connected layer and a non-linear activation function to obtain the channel weights used to adjust the importance of each semantic category; The preliminary RGB segmentation results and the preliminary depth segmentation results are concatenated along the channel dimension, and a spatial attention map is generated through a convolution operation. The spatial attention map is used to evaluate the reliability of different positions from the RGB mode and the depth mode in the spatial dimension. The channel weights are used to recalibrate the preliminary RGB segmentation result and the preliminary depth segmentation result in terms of channel dimensions. The spatial attention map is then used to perform spatial dimension weighted fusion of the two calibrated results. Finally, the results are normalized using the Softmax function to generate the second fused segmentation result.

6. The auxiliary material detection method based on 3D vision according to claim 1, characterized in that, The construction of a master decoder, inputting the first fused feature, and obtaining the first segmentation result includes: The main decoder consists of multiple cascaded decoding modules. Each decoding module sequentially upsamples the input feature map. The input of the first decoding module is the first fused feature, and the output is the feature map. The output of the previous decoding module is the input of the next decoding module, thereby increasing the spatial resolution of the feature map to the target scale. The upsampled feature map is then fused with RGB features and / or depth features from the corresponding resolution level in the dual-branch encoder. The fused feature maps are subjected to convolution and nonlinear sum operations to refine the feature representation; The last decoding module of the main decoder outputs a feature map with the same resolution as the original input image. Then, a 1x1 convolutional layer is used as a classifier to calculate the confidence score of each pixel in the feature map belonging to each semantic category. Finally, the category of each pixel is determined by the Argmax operation to generate the first segmentation result.

7. The auxiliary material detection method based on 3D vision according to claim 6, characterized in that, After performing convolution and nonlinear sum operations on the fused feature maps to refine the feature representation, the method further includes: After the output of at least one intermediate decoding module of the main decoder, an auxiliary segmentation head is added to downsample the feature map output by the intermediate decoding module to the corresponding scale and calculate the cross-entropy loss with the downsampled real label to obtain the intermediate supervision loss. The intermediate supervision loss is weighted and summed with the cross-entropy loss calculated by the first segmentation result and the real label at the final output of the main decoder to obtain the fused cross-entropy loss of the first segmentation result and the real label.

8. The auxiliary material detection method based on 3D vision according to claim 1, characterized in that, The step of locating the auxiliary material area based on the final auxiliary material segmentation result and calculating the position and height information of the auxiliary material area relative to the workpiece reference includes: Connectivity analysis was performed on the final excipient segmentation results to extract all excipient regions identified as excipients. According to the predefined reference hole positioning template, the reference hole of the workpiece is positioned in the RGB image or the depth image, and the XY pixel coordinate system and the Z-axis depth coordinate system are established with the center of the reference hole of the workpiece as the origin of the spatial coordinate system. Calculate the geometric center point of the auxiliary material area to obtain its center coordinates in the XY pixel coordinate system; Calculate the pixel distance difference between the center coordinates and the coordinates of the reference hole center in the X and Y directions; Based on the camera's intrinsic calibration data, the pixel distance difference is converted into a position offset in actual physical space to obtain the position information; The geometric center point of the auxiliary material area is mapped onto the depth image to obtain the depth value of the geometric center point, which is used as the height measurement value of the auxiliary material surface. Select at least three non-collinear points on the workpiece reference plane, obtain their depth values ​​from the depth image, and fit a reference plane using the three non-collinear points. The distance from the measured height of the auxiliary material surface to the fitting reference plane along the normal direction is calculated to obtain the height information. Based on the height information, it is determined whether the auxiliary material has a height deviation or a weight-adhesion defect.

9. The auxiliary material detection method based on 3D vision according to claim 8, characterized in that, The method further includes: The defect determination results, the corresponding RGB images and depth images, and the final auxiliary material segmentation results are stored in an online database; When the operator confirms or corrects the automatic judgment result in the system, a training sample containing the corrected label is generated. The hybrid semantic segmentation model is incrementally trained periodically using the collected training samples to optimize the model parameters.

10. A 3D vision-based auxiliary material detection device, characterized in that, The device includes: The vision inspection module is used to simultaneously acquire RGB and depth images of the workpiece to be inspected using a 3D vision sensor; The semantic segmentation module is used to input the RGB image and the depth image into a preset hybrid fusion semantic segmentation model to obtain a segmentation result image of the auxiliary material. The segmentation result image includes: a first segmentation result and a second fusion segmentation result. The auxiliary segmentation module is used to dynamically select the output of the hybrid fusion semantic segmentation model based on the data integrity of the RGB image and the depth image: when both the RGB image and the depth image are valid, the second fusion segmentation result is used as the final auxiliary segmentation result; when either the RGB image or the depth image is invalid, the first segmentation result is used as the final auxiliary segmentation result. The auxiliary material positioning module is used to locate the auxiliary material area based on the final auxiliary material segmentation result, and to calculate the position and height information of the auxiliary material area relative to the workpiece reference. The auxiliary material detection module is used to compare the position information and the height information with a preset qualified threshold to determine whether there are defects such as missing application, positional offset or over-application of the auxiliary material. The semantic segmentation module is also used to construct a dual-branch encoder, which is used to process the RGB image and the depth image respectively to obtain RGB features and depth features. The dual-branch encoder uses independent convolution kernels in the first layer and weight-shared convolution kernels in subsequent layers. A dynamic adaptive fusion module is constructed to simultaneously perform linear weighted fusion of the RGB features and the depth features to obtain a first fused feature; the RGB features and the depth features are respectively input into two independent auxiliary decoding heads to obtain a preliminary RGB segmentation result and a preliminary depth segmentation result; the preliminary RGB segmentation result and the preliminary depth segmentation result are fused by a dual attention fusion unit to obtain a second fused segmentation result; Construct a master decoder, input the first fused feature, and obtain the first segmentation result; The hybrid fusion semantic segmentation model is trained using a composite loss function, which includes at least: the cross-entropy loss between the first segmentation result and the real label, the cross-entropy loss between the second fusion segmentation result and the real label, and the feature distribution constraint loss between the RGB features and the deep features.

Images