Circular hole detection and binocular matching method considering peripheral features

Abstract

The invention discloses a circular hole detection and binocular matching method considering peripheral features. Comprising the following steps: constructing a three-dimensional part model according to structure parameters of a to-be-detected part, and constructing a data set through a rendering method by utilizing the three-dimensional part model; constructing a circular hole detection and binocular matching model, and training by using the data set to obtain a trained circular hole detection and binocular matching model; the circular hole detection and binocular matching model comprises a feature extraction backbone network, a feature encoder, two parallel feature decoders and a feature segmentation module, and the two feature decoders are respectively a detection decoder and a matching decoder; and acquiring an actual binocular image of a to-be-detected part, and inputting the actual binocular image into the trained round hole detection and binocular matching model to obtain a round hole detection result. According to the method, the position relation between the circular holes and peripheral features such as the environment can be effectively utilized, and the accuracy and robustness of binocular matching and segmentation in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to a binocular matching method, and more particularly to a method for high-precision detection, matching and segmentation of multiple similar targets in binocular images in industrial automation, robot navigation and 3D measurement scenarios. Background Technology

[0002] Binocular vision measurement is a crucial technology for achieving 3D spatial perception and geometric parameter measurement, and its primary prerequisite is the ability to accurately match features in the left and right camera images. In industrial applications, circular holes on parts are common key geometric features. However, in many real-world scenarios, parts often contain multiple circular holes with identical or highly similar geometric parameters such as size and shape.

[0003] Traditional binocular matching methods, such as template matching or matching methods based on feature descriptors like SIFT and SURF, often encounter bottlenecks when dealing with this type of problem. Template matching methods use a sliding window search, which is computationally expensive and sensitive to rotation and scale changes. Feature descriptor matching methods mainly rely on the local texture information around the feature points. When the local appearance features of multiple holes are almost indistinguishable, the discriminative power of the descriptors drops sharply, leading to a large number of mismatches.

[0004] While deep learning-based methods have achieved great success in object detection in recent years, they still present challenges when directly applied to matching similar circular holes. Most methods struggle to effectively utilize the positional relationships between the holes and the "peripheral features" formed by the target and its surrounding environment. Achieving accurate and robust matching of all holes remains a critical technical challenge when the holes are highly similar and numerous, and when complex industrial environments such as uneven lighting, reflections, and partial occlusion present interference. Summary of the Invention

[0005] This invention proposes a method for circular hole detection and binocular matching that considers peripheral features, aiming to solve the problem of inaccurate matching of circular holes in the left and right eye images when multiple circular holes with the same or similar geometric parameters exist. This method can utilize the global contextual information and relative spatial relationships of the target to achieve high-precision and robust binocular matching and segmentation of multiple similar circular holes in complex scenes.

[0006] The technical solution adopted in this invention is as follows: The method of the present invention includes the following steps: Step S1) Construct a 3D part model of the sample part, and construct a dataset by rendering the 3D part model.

[0007] In step S1, the dataset mainly consists of several sample pairs. Each sample pair includes left and right binocular images, pixel-level segmentation mask labels, and ground truth label files containing the precise pose of each circular hole.

[0008] Step S2) Construct a circular hole detection and binocular matching model, train it using the dataset, and obtain the trained circular hole detection and binocular matching model.

[0009] The circular hole detection and binocular matching model includes a feature extraction backbone network, a feature encoder, two parallel feature decoders, and a feature segmentation module. The two feature decoders are a detection decoder and a matching decoder, respectively.

[0010] The feature encoder consists of multiple stacked coding layers. Each coding layer includes a multi-scale deformable self-attention layer, a first residual connection and normalization layer, a feedforward layer, and a second residual connection and normalization layer connected in sequence, which are used to learn the complex dependencies between input feature sequences.

[0011] Both the detection decoder and the matching decoder are composed of multiple stacked decoding layers. Each decoding layer includes three functional layers: a multi-scale deformable self-attention layer, a multi-scale deformable cross-attention layer, and a feedforward layer. Each functional layer is connected to a normalization layer through a residual connection. The output of the feedforward layer is connected to a residual connection and a normalization layer.

[0012] In the decoding layer, the multi-scale deformable self-attention layer receives query information, the first residual connection and normalization layer receives the output of the multi-scale deformable self-attention layer and the query information, the multi-scale deformable cross self-attention layer receives the output of the first residual connection and normalization layer and the encoded features, the second residual connection and normalization layer receives the output of the first residual connection and normalization layer and the multi-scale deformable cross self-attention layer, the feedforward layer receives the output of the second residual connection and normalization layer, the third residual connection and normalization layer receives the output of the feedforward layer and the second residual connection and normalization layer, and the output of the third residual connection and normalization layer serves as the output of the decoding layer.

[0013] The feature segmentation module includes a multi-head attention module, a feature concatenation and semantic enhancement module, and a multi-scale feature fusion and mask prediction module. The multi-head attention module receives encoder features and decoded features output from two decoders to generate an attention map. The feature concatenation and semantic enhancement module receives the attention map and high-level semantic features output from the feature extraction backbone network to generate a feature concatenation and semantic enhancement result. The multi-scale feature fusion and mask prediction module receives the feature concatenation and semantic enhancement result and the multi-scale feature map to generate a feature segmentation result.

[0014] The training process in step S2 includes: First, the Hungarian algorithm is used to perform optimal matching between the detection results of the left eye image and the ground truth labels. The global loss is calculated based on the optimal matching results. The differential deep supervised training method is used to optimize the circular hole detection and binocular matching model that does not include the feature segmentation module. The global loss includes the category loss and bounding box loss of the left and right binocular images. Finally, the parameters of all modules in the hole detection and binocular matching model except for the feature segmentation module are frozen, and the hole detection and binocular matching model is optimized using dice loss, thereby achieving phased optimization of the feature segmentation module.

[0015] The bounding box loss is a weighted combination of L1 loss and generalized intersection-union loss.

[0016] The differentiated deep supervised training method includes: for the detection decoder with a faster convergence speed, selecting the output of the last few layers (the last 2 to 4 layers, preferably 2 layers) to calculate the loss; for the matching decoder with a slower convergence speed, calculating the loss for the output of all layers, thereby balancing the optimization process of the two parallel decoders.

[0017] Step S3) Acquire the actual left and right binocular images of the part to be detected, input them into the trained circular hole detection and binocular matching model, and obtain the segmentation mask corresponding to each of the left and right binocular images as the circular hole detection result.

[0018] Step S3 includes: Step S3.1) Obtain the left and right binocular images of the part to be detected, input them into the feature extraction backbone network, and obtain a multi-scale feature map containing information at different levels; Step S3.2) Input the multi-scale feature map into the feature encoder to obtain the encoded features; Step S3.3) The trained query vector is used as query information and input to the detection decoder along with the encoded features obtained in step S3.2 to obtain the left eye image decoding features. The left eye image detection results are generated by the feedforward neural network. The detection results include a preset number of circular hole target prediction results. Each circular hole target prediction result includes a category probability and a predicted bounding box. Step S3.4) The left eye image decoding features obtained in step S3.3 are used as query information and input into the matching decoder along with the encoded features obtained in step S3.2 to obtain the right eye image decoding features. The right eye image detection result is generated through the feedforward neural network. Step S3.5) Match the detection results of the left eye image obtained in step S3.3 and the detection results of the right eye image obtained in step S3.4, and select the successfully matched binocular image pairs as the matching results; Step S3.6) Input the multi-scale feature map obtained in step S3.1, the encoded features obtained in step S3.2, the decoded features of the left eye image obtained in step S3.3, and the decoded features of the right eye image obtained in step S3.4 into the feature segmentation module to obtain the feature segmentation result; Step S3.7) Post-process the feature segmentation result obtained in step S3.6 based on the matching result obtained in step S3.5 to obtain the prediction segmentation mask.

[0019] The beneficial effects of this invention are: 1) Introducing a deformable attention-based encoder-decoder architecture enables the model to learn and utilize the global context and peripheral features of the target, improving the detection and matching accuracy in scenes with dense similar targets.

[0020] 2) The designed binocular feature matching module based on query vectors uses the detection results of the left image as the guiding information for matching the right image, and directly establishes the matching relationship through the index, which simplifies the matching process and makes the method efficient.

[0021] 3) The proposed differentiated deep supervision training strategy for parallel architecture helps to solve the problem of unbalanced convergence of dual decoders, making the model training process more stable and helping to improve overall performance.

[0022] In summary, this invention can effectively utilize the positional relationship between the circular holes and external features such as the environment to improve the accuracy and robustness of binocular matching and segmentation in complex scenarios. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of the circular hole detection and binocular vision matching model that considers peripheral features in this invention.

[0024] Figure 2 This is a schematic diagram of the encoder-decoder architecture based on multi-scale deformable attention in this invention.

[0025] Figure 3 This is a detailed schematic diagram of the circular hole feature detection module based on deformable attention in this invention.

[0026] Figure 4 This is a detailed schematic diagram of the binocular feature matching module based on query vector in this invention.

[0027] Figure 5 This is a schematic diagram of the binocular feature matching results in this invention, showing the matching between the predicted bounding box and the real label.

[0028] Figure 6 This is a schematic diagram of the architecture of the circular hole feature segmentation module in this invention.

[0029] Figure 7This is a schematic diagram of the process of generating synthetic data using 3D rendering software in this invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] This invention discloses a circular hole detection and binocular matching method considering peripheral features. The method includes: constructing a circular hole feature detection module based on deformable attention to extract multi-scale features of the circular hole itself and its surrounding environment from binocular images; constructing a binocular feature matching module based on a query vector, using the detection result of the left image as query information to establish feature associations between the left and right images; and finally, constructing a feature segmentation module to generate pixel-level segmentation masks for successfully matched circular holes. This invention also designs a loss function for parallel architecture and a differentiated deep supervised learning strategy to improve model performance.

[0032] The method of the present invention specifically includes the following steps: Step S1) Construct a 3D part model of the sample part, and construct a dataset by rendering the 3D part model.

[0033] The parts include at least one surface with a circular hole structure, and the materials used for the parts include, but are not limited to, plastics and metals. The parts include, but are not limited to, mechanical base components such as round / square flanges and end caps, as well as electrical connectors such as electric vehicle charging heads.

[0034] The dataset mainly consists of several sample pairs. Each sample pair includes left and right binocular images, pixel-level segmentation mask labels, and ground truth label files containing the precise pose of each hole.

[0035] Alternatively, a dataset can be constructed using rendering methods with the help of 3D rendering software.

[0036] The above-mentioned optional implementation scheme can be achieved through the following process: by randomly laying out the 3D part model in the virtual scene, setting diverse lighting and camera poses, generating synthetic data containing binocular images, segmentation masks and ground truth labels, and using methods such as block erasure, random cropping and random background to perform data augmentation processing on the synthetic data to improve the generalization ability of the model.

[0037] Step S2) Construct a circular hole detection and binocular matching model, train it using the dataset, and obtain the trained circular hole detection and binocular matching model.

[0038] The circular hole detection and binocular matching model includes a feature extraction backbone network, a feature encoder, two parallel feature decoders, and a feature segmentation module. The two feature decoders are a detection decoder and a matching decoder.

[0039] In the circular hole detection and binocular matching model, the feature extraction backbone network is used to extract multi-scale feature maps containing rich semantic information and spatial details. A pre-trained convolutional neural network can be used. Here, pre-training refers to the process of training on a large-scale general dataset. A pre-trained convolutional neural network is a visual feature extractor that already has general and transferable feature representation capabilities.

[0040] In the circular hole detection and binocular matching model, the feature encoder employs a multi-scale deformable attention layer as its core functional layer to encode features, learn the circular hole features, the relationship between the circular hole and the background, and the relative positional relationships between the circular holes, and output encoded features. This encoder can adaptively perform sparse sampling on the feature map, without being limited by the local receptive field of traditional convolutional kernels, thus effectively learning the circular hole target itself, the target and the background, and the long-distance dependencies and spatial relative positional relationships between different circular holes, generating encoded features with rich contextual information.

[0041] Specifically, the feature encoder consists of multiple stacked coding layers. Each coding layer includes a multi-scale deformable self-attention layer, a first residual connection and normalization layer, a feedforward layer, and a second residual connection and normalization layer connected in sequence, which are used to learn the complex dependencies between the input feature sequences.

[0042] In the circular hole detection and binocular matching model, both the detection decoder and the matching decoder consist of multiple stacked decoding layers. The detection decoder receives encoded features and a set of learnable query vectors representing the target object. Through a multi-layer cross-attention mechanism, it decodes the information in the encoded features to obtain the left-eye image decoded features. These features are then passed through a feedforward neural network to generate the left-eye image detection result, which includes the class confidence score and bounding box coordinates for each predicted circular hole. The matching decoder runs in parallel with the detection decoder. Instead of using the learnable query vectors, the matching decoder uses the left-eye image decoded features (which have already learned information from the left image) output from the last layer of the detection decoder as the query information, inputting it along with the encoded features into the matching decoder. In this way, the detection result from the left image is used as a guide to find corresponding features in the right image. The matching decoder outputs the right-eye image decoded features. Since its input query and output result have a one-to-one correspondence in the index, it directly completes the matching of predicted circular holes in the left and right eye images.

[0043] Specifically, each decoding layer includes three functional layers: a multi-scale deformable self-attention layer, a multi-scale deformable cross-attention layer, and a feedforward layer. Each functional layer is connected to a normalization layer via a residual connection, and the output of the feedforward layer is connected to a normalization layer via a residual connection.

[0044] More specifically, in the decoding layer, the multi-scale deformable self-attention layer receives query information, the first residual connection and normalization layer receives the output of the multi-scale deformable self-attention layer and the query information, the multi-scale deformable cross self-attention layer receives the output of the first residual connection and normalization layer and the encoded features output by the feature encoder, the second residual connection and normalization layer receives the outputs of the first residual connection and normalization layer and the multi-scale deformable cross self-attention layer, the feedforward layer receives the output of the second residual connection and normalization layer, the third residual connection and normalization layer receives the outputs of the feedforward layer and the second residual connection and normalization layer, and the output of the third residual connection and normalization layer is used as the output of the decoding layer, input to the next decoding layer or output from the corresponding decoder (detection decoder / matching decoder).

[0045] In the circular hole detection and binocular matching model, the decoded features output by the detection decoder are used as the query information input for the matching decoder. This allows the target features, background features, and relative position features learned from the left eye image to guide the feature detection and matching process of the right eye image.

[0046] In the circular hole detection and binocular matching model, after detection and matching are completed, the feature segmentation module receives the output of the feature extraction backbone network, encoder features, and decoded features from the two decoders to obtain the feature segmentation result. The feature segmentation result and the matching results of the left and right eye images are then input into the post-processing module to obtain the predicted segmentation mask. The feature segmentation module and the post-processing module focus on the target region through a multi-head attention mechanism, outputting a high-precision pixel-level segmentation mask for each successfully matched circular hole target, providing shape priors for subsequent applications.

[0047] Specifically, the feature segmentation module includes a multi-head attention module, a feature concatenation and semantic enhancement module, and a multi-scale feature fusion and mask prediction module. The multi-head attention module receives encoder features and decoded features from the two decoders to generate an attention map. The feature concatenation and semantic enhancement module receives the attention map and high-level semantic features from the feature extraction backbone network to generate a feature concatenation and semantic enhancement result. The multi-scale feature fusion and mask prediction module receives the feature concatenation and semantic enhancement result and the multi-scale feature map to generate a feature segmentation result.

[0048] Specifically, the post-processing module receives the feature segmentation results and the matching results of the left and right eye images, and generates a predictive segmentation mask.

[0049] The training process includes: First, the Hungarian algorithm is used to perform optimal matching between the detection results of the left eye image and the ground truth labels. The global loss is calculated based on the optimal matching results. The differential deep supervised training method is used to optimize the circular hole detection and binocular matching model that does not include the feature segmentation module. Finally, the parameters of all modules in the hole detection and binocular matching model except for the feature segmentation module are frozen, and the hole detection and binocular matching model is optimized using dice loss, thereby achieving phased optimization of the feature segmentation module.

[0050] Preferably, the loss includes class loss for the left and right binocular images and bounding box loss. The bounding box loss is preferably a weighted combination of L1 loss and generalized intersection-union (GioU) loss.

[0051] More preferably, the process of calculating the global loss includes: the number of predicted results output by the model is usually greater than the number of real targets. Using the Hungarian algorithm, the optimal arrangement with the minimum matching loss is found between the detection result of the left eye image and the real label, that is, the optimal matching scheme with the minimum total cost, by calculating the matching loss. After finding the optimal match, the class loss and bounding box loss of the left and right images are calculated simultaneously for the predicted-label pairs that are successfully matched in the optimal arrangement to obtain the global loss.

[0052] The differentiated deep supervised training method includes: for the detection decoder with a faster convergence speed, the loss is calculated from the output of the last few layers (the last 2 to 4 layers, preferably 2 layers; in specific implementation, the output can be set according to experience or adaptively adjusted according to the training results); for the matching decoder with a slower convergence speed, the loss is calculated from the output of all layers, so as to balance the optimization process of the two parallel decoders.

[0053] Preferably, the dice loss is the loss between the feature segmentation result and the real mask.

[0054] Step S3) Acquire the actual left and right binocular images of the part to be detected, input them into the trained circular hole detection and binocular matching model, and obtain the segmentation mask corresponding to each of the left and right binocular images as the circular hole detection result.

[0055] Specifically, step S3 includes: Step S3.1) Obtain the left and right binocular images of the part to be detected, and input the left and right binocular images into the feature extraction backbone network to obtain a multi-scale feature map containing information at different levels; Step S3.2) Input the multi-scale feature map into the feature encoder for feature encoding, learn the features of the circular holes, the relationship between the circular holes and the background, and the relative positional relationship between the circular holes, and obtain the encoded features; Step S3.3) The trained query vector is used as query information and input into the detection decoder along with the encoded features obtained in step S3.2 to obtain the left eye image decoding features. The left eye image decoding features are used to generate the left eye image detection result through a feedforward neural network. The detection results include a preset number of circular hole target prediction results, and each circular hole target prediction result includes a predicted circular hole class probability and a predicted bounding box; Step S3.4) The left eye image decoding features obtained in step S3.3 are used as query information and input into the matching decoder along with the encoded features obtained in step S3.2 to obtain the right eye image decoding features. The right eye image decoding features are then used by a feedforward neural network to generate the right eye image detection result. The detection results include a preset number of circular hole target prediction results, and each circular hole target prediction result includes a class probability and a predicted bounding box; Step S3.5) Match the detection results of the left eye image obtained in step S3.3 and the detection results of the right eye image obtained in step S3.4, and select the successfully matched binocular image pairs as the matching results; Specifically, this step involves: based on the index correspondence between the input (left eye image decoding features) and output (right eye image decoding features) of the matching decoder, predicting circular hole targets with the same index in the left and right binocular images, comparing the class probability in the predicted circular hole target in the left eye image with a preset confidence threshold, and comparing the class probability in the predicted circular hole target in the right eye image with a preset matching confidence threshold. If the class probability in the predicted i-th circular hole target in the left eye image is greater than the preset confidence threshold, and the class probability in the predicted i-th circular hole target in the right eye image is greater than the preset matching confidence threshold, then the match is considered successful; otherwise, the match is considered unsuccessful.

[0056] Step S3.6) Input the multi-scale feature map obtained in step S3.1, the encoded features obtained in step S3.2, the decoded features of the left eye image obtained in step S3.3, and the decoded features of the right eye image obtained in step S3.4 into the feature segmentation module to obtain the feature segmentation result; The feature segmentation results include the segmentation mask corresponding to each predicted circular hole in the binocular image; Step S3.7) Then, based on the matching results obtained in step S3.5, the feature segmentation results obtained in step S3.6 are post-processed to obtain the bounding box coordinates of each predicted circular hole in the left and right binocular images and the final pixel-level segmentation mask.

[0057] The embodiments of the present invention and their implementation process are as follows: Example In this embodiment, the matching process includes the following steps: Step S1) Dataset Construction and Preprocessing. To obtain sufficient and diverse training data, this embodiment uses the BlenderProc 3D rendering tool to generate a synthetic dataset. The process is as follows: Figure 7 As shown: First, import the CAD model of the part to be inspected into the 3D scene; Secondly, the parts are automatically translated and rotated randomly using scripts, and collision detection is used to ensure the physical realism of the layout; Then, randomized light sources (type, position, intensity) and binocular camera poses are set in the scene to simulate various complex industrial environments; Finally, the rendering generates a stereo image, a pixel-level segmentation mask, and a ground truth label file containing the precise pose of each circular hole.

[0058] To further improve the robustness of the model, data augmentation is performed on the generated training data, including: block erasure (randomly occluding local areas of the image), random cropping, and random background fusion.

[0059] Step S2) Construct and train the circular hole detection and binocular matching model. The overall architecture of the circular hole detection and binocular matching model is as follows: Figure 1 As shown, it mainly consists of a shared feature extraction backbone network, a feature encoder, two parallel feature decoders (a detection decoder and a matching decoder), and a feature segmentation module. Specifically, it includes the following steps: S2.1) Construct a circular hole detection and binocular matching model.

[0060] S2.1.1) Construct a feature detection module based on deformable attention. This module is responsible for detecting circular holes in the left camera image and providing information for subsequent matching. Its architecture is as follows: Figure 3 As shown: First, a ResNet-50 network is used as the feature extraction backbone. The left and right binocular images are input into the feature extraction backbone, and the outputs of stages 3, 4, and 5 are extracted. These outputs are then fused by the feature fusion module to obtain a multi-scale feature map x. m , where m represents the number of scales. In this embodiment, feature fusion includes processes such as 1×1 convolution and splicing.

[0061] Subsequently, the multi-scale feature map x m Input a feature encoder consisting of 6 cascaded coding layers stacked together (e.g.) Figure 2 (As shown). Each coding layer core employs a multi-scale deformable self-attention module. This module can dynamically determine the sampling point position based on the input features, thereby effectively capturing long-distance dependencies and spatial relationships, and learning rich peripheral features; The encoded features output by the feature encoder, along with a set of learnable query vectors representing the "object queries," are fed into a detection decoder consisting of six decoding layers. The detection decoder integrates information from the encoded features into the query vectors using a cross-attention mechanism, ultimately outputting decoded features. These decoded features are then processed by a feedforward neural network (FFN) to output the left-eye image detection results, i.e., N predicted holes, where the i-th predicted hole contains the class probability. and bounding box coordinates , where l represents the left eye image.

[0062] S2.1.2) Construct a binocular feature matching module based on query vectors. This module is responsible for finding targets in the right-eye image that match the circular holes detected in the left-eye image. Its architecture is as follows: Figure 4 As shown.

[0063] The binocular feature matching module and the feature detection module share the same encoded features output by the feature encoder. The key difference is that it does not use a learnable query vector; instead, it directly uses the decoded features of the left-eye image output by the detection decoder as its input query. The features output by the detection decoder have already encoded all the information (including its own features and surrounding features) for each detected hole in the left image. This information is then used as the query and input into the matching decoder.

[0064] The matching decoder also consists of 6 decoding layers. After decoding, the right eye image decoding features are obtained, and then the right eye image detection results, namely N predicted holes, are output through FFN. Each predicted hole contains the class probability. and bounding box coordinates , where r represents the right eye image.

[0065] Since the indices for the input query and the output prediction are strictly corresponding, if the i-th prediction in the left graph is considered a valid target, then the i-th prediction in the right graph is its matching result.

[0066] The number of prediction results in the left and right eye images is the same preset value.

[0067] S2.2) Construct a circular hole detection and binocular matching model.

[0068] S2.2.1) Design the loss function.

[0069] To select the predictions that correspond to the K ground truths from N predictions and to supervise their selection, this invention employs the Hungarian algorithm for optimal matching. Matching cost function. The calculation, performed only on the left graph, combines the negative log-likelihood loss from the class prediction with the loss from the bounding box, as shown below. Figure 5 As shown.

[0070] In the formula, It is the matching cost function. It is an indicator function, if the true category label If the target is not an empty category (i.e., the target exists), the value is 1; otherwise, the value is 0. It is the prediction result Category The predicted probability, The bounding box loss is a weighted combination of L1 loss and generalized intersection-union (GUUID) loss, and it is scale-invariant. The GUUID loss measures the overlap and shape difference between two bounding boxes, while the weights of both L1 and GUUID losses are hyperparameters. `i` is the predicted index, and `l` represents the left eye image.

[0071] Find the optimal matching permutation Then, calculate the final model loss function. The loss function is supervised simultaneously on both the left and right graphs for all successfully matched pairs: In the formula, This is the model loss function, where i is the predicted index, l represents the left eye image, and r represents the right eye image. After optimal matching, the th The prediction results correspond to the true category in the left eye image. The logarithm of the predicted probability, After optimal matching, in the left eye image, the first... The weighted loss (L1 loss, generalized intersection-over-union ratio) between the bounding boxes of each real target and their corresponding predicted bounding boxes. After optimal matching, the th The prediction results correspond to the true category in the right eye image. The logarithm of the predicted probability, After optimal matching, in the right eye image, the first... The weighted loss between the bounding boxes of real targets and their corresponding predicted bounding boxes (L1 loss, generalized intersection-union ratio).

[0072] S2.2.2) Differentiated Deep Supervision Training: During training, both the detection decoder and the matching decoder employ deep supervision, meaning the output of each decoding layer is used to calculate the loss. However, due to the relative simplicity of the detection task, its decoder converges faster. To prevent it from dominating the training process and suppressing the learning of the matching decoder, this invention employs a differentiated strategy: for the detection decoder, only the output of the last two layers is used to calculate the loss; while for the matching decoder, the output of all six layers is used to calculate the loss. This effectively balances the gradients of the two parallel branches, promoting an overall performance improvement.

[0073] S2.3) Constructing the circular hole feature segmentation module: After detection and matching are completed, the model enters the final circular hole feature segmentation stage, such as... Figure 6 As shown.

[0074] The circular hole feature segmentation module receives multi-scale features from the feature extraction backbone network, features output by the encoder, and decoded features output by the two decoders. The decoded features from the two decoders are then processed by the multi-head attention module to generate an attention map.

[0075] By performing multi-level upsampling and feature fusion in the multi-scale feature fusion and mask prediction module, and finally predicting the segmentation mask of each detected circular hole by the last convolutional layer of the multi-scale feature fusion and mask prediction module, the feature segmentation result is obtained.

[0076] Figure 6 In this context, high-level semantic features refer to the output of semantic features from later stages of ResNet-50 (feature extraction backbone network) after convolution operations.

[0077] The circular hole feature segmentation module is trained using Dice loss alone, with all other parameters of the model frozen during training. Dice loss is the difference between the feature segmentation result and the ground truth mask.

[0078] S2.4) Constructing the post-processing module: The post-processing module first upsamples the predicted mask to the original image size through bilinear interpolation, then binarizes it with a fixed threshold and combines it with morphological closing operations to eliminate small holes; then spatially aligns the binary mask with the predicted detection boxes (the pairs of successfully detected and matched circular holes in the left and right eye images), and then removes abnormal masks with an intersection-union ratio (IU) of less than 0.3 with the effective predicted boxes. Finally, it outputs the optimized instance segmentation result, which is the pixel-level segmentation mask of each successfully detected and matched circular hole.

[0079] The goal of this stage is to Step S3) Obtain the detection result based on the actual binocular image of the part to be inspected. The workflow is as follows: S3.1) Input: Obtain the left and right viewpoint binocular image pairs to be processed.

[0080] S3.2) Feature Extraction and Encoding: The stereo image pairs are input into the pre-trained model. The images first pass through a shared feature extraction backbone network to generate multi-scale feature maps. Subsequently, these feature maps are fed into the feature encoder to generate encoded features containing rich contextual information.

[0081] S3.3) Left image detection: The features output by the encoder and a set of fixed (trained) query vectors are fed into the detection decoder. The detection decoder outputs the left image decoding features, which are then passed through a feedforward neural network to output N sets of prediction results for the left image. Each set of results includes a class confidence and a bounding box.

[0082] S3.4) Left Image Target Filtering: Set a preset confidence threshold (e.g., 0.7). Iterate through all prediction results in the left image, and only when the confidence of a prediction result is higher than the threshold is it determined as a successfully detected valid target.

[0083] S3.5) Right Image Matching: The features output by the encoder and the decoded features output by the detection decoder (as queries) are fed into the matching decoder. The matching decoder also outputs the right image decoded features, which are then passed through the feedforward neural network to output N sets of prediction results for the right image.

[0084] S3.6) Matching relationship determination: For each target judged as valid in the left image (its index is i), examine its corresponding prediction result in the right image (both index is i). Set a preset matching confidence threshold. If the confidence of the i-th prediction in the right image exceeds the threshold, the i-th target in both the left and right images is considered to be successfully matched; otherwise, it is considered a failed match.

[0085] S3.7) Segmentation Mask Generation: The backbone network features, encoder features, and decoded features of the two images are fed into the feature segmentation module to generate segmentation masks for each detected circular hole. The segmentation masks for each detected circular hole, along with all successfully detected and matched circular hole targets, are input into the post-processing module. After post-processing, the final pixel-level segmentation mask is obtained.

[0086] S3.8) Output: Finally, the model outputs the bounding box coordinates and pixel-level segmentation masks of all successfully matched hole pairs in their respective images.

[0087] In summary, this invention, through a complete design, from model architecture and loss function to training strategy, systematically solves the problem of binocular matching and segmentation of similar circular holes in complex scenes.

[0088] The above specific embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for circular hole detection and binocular matching considering peripheral features, characterized in that, Includes the following steps: Step S1) Construct a 3D part model of the sample part, and construct a dataset by rendering the 3D part model; Step S2) Construct a circular hole detection and binocular matching model, train it using the dataset, and obtain the trained circular hole detection and binocular matching model; The circular hole detection and binocular matching model includes a feature extraction backbone network, a feature encoder, two parallel feature decoders, and a feature segmentation module. The two feature decoders are a detection decoder and a matching decoder, respectively. Step S3) Acquire the actual left and right binocular images of the part to be detected, input them into the trained circular hole detection and binocular matching model, and obtain the segmentation mask as the circular hole detection result.

2. The circular hole detection and binocular matching method considering peripheral features according to claim 1, characterized in that: In step S1, the dataset mainly consists of several sample pairs. Each sample pair includes left and right binocular images, segmentation mask labels, and ground truth labels containing the pose of each circular hole.

3. The circular hole detection and binocular matching method considering peripheral features according to claim 1, characterized in that: Step S3 includes: Step S3.1) Obtain the left and right binocular images of the part to be detected, input them into the feature extraction backbone network, and obtain a multi-scale feature map containing information at different levels; Step S3.2) Input the multi-scale feature map into the feature encoder to obtain the encoded features; Step S3.3) The trained query vector is used as query information and input to the detection decoder along with the encoded features obtained in step S3.2 to obtain the left eye image decoding features. The left eye image detection results are generated by the feedforward neural network. The detection results include a preset number of circular hole target prediction results. Each circular hole target prediction result includes a category probability and a predicted bounding box. Step S3.4) The left eye image decoding features obtained in step S3.3 are used as query information and input into the matching decoder along with the encoded features obtained in step S3.2 to obtain the right eye image decoding features. The right eye image detection result is generated through the feedforward neural network. Step S3.5) Match the detection results of the left eye image obtained in step S3.3 and the detection results of the right eye image obtained in step S3.4, and select the successfully matched binocular image pairs as the matching results; Step S3.6) Input the multi-scale feature map obtained in step S3.1, the encoded features obtained in step S3.2, the decoded features of the left eye image obtained in step S3.3, and the decoded features of the right eye image obtained in step S3.4 into the feature segmentation module to obtain the feature segmentation result; Step S3.7) Post-process the feature segmentation result obtained in step S3.6 based on the matching result obtained in step S3.5 to obtain the prediction segmentation mask.

4. The circular hole detection and binocular matching method considering peripheral features according to claim 1 or 3, characterized in that: The feature encoder is composed of multiple stacked coding layers. Each coding layer includes a multi-scale deformable self-attention layer, a first residual connection and normalization layer, a feedforward layer, and a second residual connection and normalization layer connected in sequence.

5. The circular hole detection and binocular matching method considering peripheral features according to claim 1 or 3, characterized in that: Both the detection decoder and the matching decoder are composed of multiple stacked decoding layers. Each decoding layer includes three functional layers: a multi-scale deformable self-attention layer, a multi-scale deformable cross-attention layer, and a feedforward layer. Each functional layer is connected to a normalization layer through a residual connection. The output of the feedforward layer is connected to a residual connection and a normalization layer.

6. The circular hole detection and binocular matching method considering peripheral features according to claim 5, characterized in that: In the decoding layer, the multi-scale deformable self-attention layer receives query information, the first residual connection and normalization layer receives the output of the multi-scale deformable self-attention layer and the query information, the multi-scale deformable cross self-attention layer receives the output of the first residual connection and normalization layer and the encoded features, the second residual connection and normalization layer receives the output of the first residual connection and normalization layer and the multi-scale deformable cross self-attention layer, the feedforward layer receives the output of the second residual connection and normalization layer, the third residual connection and normalization layer receives the output of the feedforward layer and the second residual connection and normalization layer, and the output of the third residual connection and normalization layer serves as the output of the decoding layer.

7. The circular hole detection and binocular matching method considering peripheral features according to claim 1, characterized in that: The feature segmentation module includes a multi-head attention module, a feature concatenation and semantic enhancement module, and a multi-scale feature fusion and mask prediction module; the multi-head attention module receives encoder features and decoded features output by two decoders to generate an attention map; The feature concatenation and semantic enhancement module receives the attention map and the high-level semantic features output by the feature extraction backbone network, and generates the feature concatenation and semantic enhancement results; The multi-scale feature fusion and mask prediction module receives the feature concatenation and semantic enhancement results and the multi-scale feature map, and generates the feature segmentation result.

8. The circular hole detection and binocular matching method considering peripheral features according to claim 1, characterized in that: The training process in step S2 includes: First, the Hungarian algorithm is used to perform optimal matching between the detection results of the left eye image and the ground truth labels. The global loss is calculated based on the optimal matching results. The differential deep supervised training method is used to optimize the circular hole detection and binocular matching model that does not include the feature segmentation module. The global loss includes the category loss and bounding box loss of the left and right binocular images. Finally, the parameters of all modules in the hole detection and binocular matching model except for the feature segmentation module were frozen, and the hole detection and binocular matching model was optimized using dice loss.

9. The circular hole detection and binocular matching method considering peripheral features according to claim 7, characterized in that: The differentiated deep supervised training method includes: for the detection decoder, calculating the loss by selecting the output of the last few layers; for the matching decoder, calculating the loss by calculating the output of all layers, thereby balancing the optimization process of the two parallel decoders.

10. The circular hole detection and binocular matching method considering peripheral features according to claim 7, characterized in that: The bounding box loss is a weighted combination of L1 loss and generalized intersection-union loss.

Images