A complex assembly scene anomaly detection method and device based on virtual-real comparison

Abstract

The application provides a complex assembly scene anomaly detection method and device based on virtual-real comparison, which comprises the following steps: S100, collecting a virtual image and a real image, and extracting key point positions and feature descriptors to realize high-precision geometric registration of the virtual image and the real image; S200, based on the feature descriptors corresponding to the key point positions, performing multi-head feature correlation analysis and adaptive threshold segmentation on the registered virtual image and real image to obtain an anomaly detection mask; and S300, based on the anomaly detection mask, performing coarse-to-fine screening on an abnormal area based on a SAM model to realize accurate determination and contour segmentation of assembly anomalies.

Description

Technical Field

[0001] This invention belongs to the field of anomaly detection technology, specifically relating to an anomaly detection method and device for complex assembly scenarios based on virtual-real comparison. Background Technology

[0002] In modern high-end equipment manufacturing fields, such as aerospace, precision instruments, and the automotive industry, the final quality and reliability of products depend heavily on the precision and consistency of their assembly processes. Complex assembly scenarios typically involve tens of thousands of parts, follow strict process sequences, and involve human-machine collaboration and complex spatial interactions. Any minute geometric deviation, process error, or incorrect / missing part assembly can lead to performance degradation or even functional failure of the entire product. Therefore, achieving comprehensive and intelligent anomaly detection in complex assembly scenarios is a key technological link in ensuring product quality, improving production efficiency, and promoting the implementation of intelligent manufacturing.

[0003] Traditional quality inspection methods heavily rely on experienced quality inspectors for manual visual inspection, which is susceptible to subjective fatigue, resulting in low efficiency and difficulty in standardization. In recent years, computer vision-based automatic detection technologies have made significant progress, especially deep learning models, which have demonstrated powerful capabilities in image classification and object detection. However, when these methods are directly applied to complex and dynamic real-world assembly environments, they still face serious challenges: First, supervised learning models heavily rely on large amounts of labeled data, while "abnormal" samples in industrial production are scarce and varied in form, leading to high data collection and labeling costs; second, real-world workshop environments have many uncertainties such as lighting fluctuations, changing viewpoints, and partial occlusion, making models relying solely on the appearance features of two-dimensional images lack generalization ability and robustness; third, deeper assembly logic errors (such as reversed order) and violations of inter-component relationships are difficult to effectively capture through isolated image analysis. To overcome these difficulties, a method for anomaly detection in complex assembly scenarios based on virtual-real comparison is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a method and device for detecting anomalies in complex assembly scenarios based on virtual-real comparison, in order to overcome problems such as data dependence and environmental factors during the assembly quality inspection process.

[0005] This invention is mainly achieved through the following technical solutions: Firstly, this application provides an anomaly detection method for complex assembly scenarios based on virtual-real comparison, the method comprising: Step S100: Acquire virtual and real images, extract key point positions and feature descriptors to achieve high-precision geometric registration between virtual and real images; Step S200: Based on the feature descriptors corresponding to the key point locations, perform multi-head feature correlation analysis and adaptive threshold segmentation on the registered virtual image and the real image to obtain an anomaly detection mask. ; Step S300: Based on anomaly detection mask Based on the SAM model, abnormal areas are screened from coarse to fine to achieve accurate judgment and contour segmentation of assembly abnormalities.

[0006] Specifically, step S100 includes: Step S101: Collect virtual and real images of the same target object, and use them as inputs for virtual and real images respectively; use the SuperPoint network model based on self-supervised deep learning to preprocess the grayscale values ​​of the virtual and real images to obtain the corresponding single-channel grayscale images; Step S102: Input the virtual image and the real image after grayscale preprocessing into the shared encoder respectively, and extract the shared feature maps corresponding to the two; Step S103: Input the shared feature map into the key point decoder and the feature descriptor decoder. Output the pixel-level key point positions through the key point decoder and output the feature descriptor corresponding to the key point positions through the feature descriptor decoder.

[0007] Specifically, step S200 includes: Step S201: Input the virtual image and the real image into the image encoding network to extract multi-layer deep features; combine the key point positions and their corresponding feature descriptors to perform association constraints on the deep features to enhance the semantic consistency between corresponding regions across domains; Step S202: Based on the multi-layer deep features, calculate the comprehensive similarity response map between the virtual image and the real image in the multi-head attention space. To characterize the feature correlation between virtual and real images under different attention heads; Step S203: Based on the comprehensive similarity response map mean Standard deviation Total number of pixels , No. Similarity value at each pixel position Using the formula Calculate the skewness of the similarity distribution ; Step S204: Based on skewness Using the formula Dynamically adjust the binarization threshold ,in, This serves as a baseline threshold to provide the basic segmentation level. This is a sensitivity factor used to control the impact of skewness changes on the threshold adjustment range; This is a correction term used to compensate for system offsets caused by different image scenes, different coding layer feature distributions, or different data acquisition conditions, thereby improving the stability and adaptability of threshold setting; Step S205: Combine the similarity response map With binarization threshold Comparison to generate an anomaly detection mask .

[0008] Specifically, step S202 includes: Step S2021: Based on the key feature vector of the virtual image Key feature vectors of real images Calculate the corresponding cosine similarity matrix. :

[0009] in, and These represent the virtual image feature location index and the real image feature location index, respectively; For virtual images in the first Layer The first thing to pay attention to. Key feature vectors at each position; For the real image in the first Layer The first thing to pay attention to. Key feature vectors at each position; Step S2022: Obtain the cosine similarity matrix from multiple attention heads Weighted fusion or linear mapping is performed to obtain a comprehensive similarity response map. , represented as:

[0010] in, To focus on the number of heads, For the first The weighting coefficient of each attention head. This is a linear layer or mapping function used to fuse multi-head similarity results into a single-channel similarity response map.

[0011] Specifically, step S205 includes: When the similarity value at a certain position in the comprehensive similarity response graph is less than the binarization threshold When the similarity value at a given location is greater than or equal to the binarization threshold, the location is determined to be an abnormal pixel region. When this occurs, the pixel at that location is determined to be a normal region pixel; thus, an anomaly detection mask is obtained. Its expression is:

[0012] in, Indicates the first This location belongs to the assembly anomaly area. Indicates the first These locations are within the normal range.

[0013] Specifically, step S300 includes: Step S301: Input the virtual image into the full-image segmentation mode of the SAM model to generate a set of category-independent candidate part masks. ; Step S302: Calculate the mask for each candidate part With anomaly detection mask overlap rate :

[0014] in, This represents the number of pixels in the intersection region between the candidate part mask and the initial anomalous mask. This represents the total number of pixels in the candidate part mask; a geometric threshold is set. , retain satisfaction The candidate regions are used as candidate anomaly objects; Step S303: Perform semantic similarity matching on candidate abnormal objects, and extract the Mask Embedding of the virtual image and the real image within the corresponding mask regions, denoted as... and And calculate the cosine similarity. :

[0015] in, Used to characterize the semantic consistency of candidate regions between virtual and real images; Step S304: Combine geometric overlap rate and semantic similarity The system performs decision-making and screening on candidate anomalies, retaining geometrically consistent regions with significant semantic differences as the final anomaly regions, and outputs the anomaly contour segmentation results.

[0016] Specifically, step S304 includes: calculating the anomaly determination score of the candidate anomaly object. :

[0017] in, To integrate the weighting coefficients, and ; Used to characterize the geometric consistency between the candidate region and the anomaly detection mask; Used to characterize the semantic consistency of candidate regions between virtual and real images; Used to characterize the degree of semantic difference in candidate regions.

[0018] By setting an anomaly detection threshold , retain satisfaction The candidate regions are selected as the final anomaly regions, and the anomaly contour segmentation results are output.

[0019] Secondly, this application provides an anomaly detection device for complex assembly scenarios based on virtual-real comparison, the device being used to implement the above-mentioned anomaly detection method for complex assembly scenarios based on virtual-real comparison.

[0020] The beneficial effects of this invention are as follows: This invention proposes an anomaly detection method for complex assembly scenarios based on virtual-real data comparison. It achieves high-precision alignment of virtual and real data through self-supervised deep learning, and accurately identifies assembly anomalies by combining multi-layer feature comparison and a geometric-semantic dual refinement strategy. This method is particularly suitable for anomaly detection needs in complex assembly scenarios, effectively improving the robustness and accuracy of anomaly detection. It solves the problems of insufficient capture of deep semantic associations, poor threshold adaptability, and weak resistance to noise interference in traditional methods, ensuring the quality and efficiency of complex equipment assembly. Attached Figure Description

[0021] Figure 1 This is a flowchart of an anomaly detection method for complex assembly scenarios based on virtual-real comparison, provided by an embodiment of the present invention.

[0022] Figure 2 This is the dataset constructed in the embodiments of the present invention.

[0023] Figure 3 This is a schematic diagram of the virtual-real image alignment process according to an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the multi-layer feature comparison strategy according to an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the abnormal score fusion and decision-making process according to an embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] Example 1 This application provides an anomaly detection method for complex assembly scenarios based on virtual-real comparison, including the following steps: Step S100: Acquire virtual and real images, extract key point locations and feature descriptors, and achieve high-precision geometric registration between virtual and real images.

[0028] Specifically, step S100 includes: Step S101: Collect virtual and real images of the same target object, and use them as inputs for virtual and real images respectively; use the SuperPoint network model based on self-supervised deep learning to preprocess the grayscale values ​​of the virtual and real images to obtain the corresponding single-channel grayscale images.

[0029] Preferably, the grayscale image is further normalized to meet the network input requirements.

[0030] The SuperPoint network model based on self-supervised deep learning employs a VGG-like architecture shared encoder to perform multi-layer convolutional feature extraction on the input grayscale image, thereby obtaining deep semantic features that characterize the local texture information and overall structural information of the image.

[0031] Step S102: Input the virtual image and the real image after grayscale preprocessing into the shared encoder respectively, and extract the shared feature maps corresponding to the two; It should be noted that the shared feature map is used for both keypoint detection and local feature description simultaneously.

[0032] Step S103: Input the shared feature map into the key point decoder and the feature descriptor decoder. Output the pixel-level key point positions through the key point decoder and output the feature descriptor corresponding to the key point positions through the feature descriptor decoder.

[0033] In practical applications, feature descriptors can be 256-dimensional feature descriptors.

[0034] The key point location is the two-dimensional pixel coordinate in the image plane coordinate system. Alternatively, the location of a salient point can be determined by thresholding and non-maximum suppression in the keypoint response map; the keypoint location is used to characterize stable and repeatable local salient regions in the image and to provide positional constraints for feature matching and geometric transformation estimation between the virtual image and the real image.

[0035] The feature descriptor is a 256-dimensional vector representation that corresponds one-to-one with the location of each key point, used to characterize the local appearance and structural features of the key point's neighborhood; preferably, the feature descriptor is normalized to improve the consistency of feature representation under different viewpoints and lighting conditions; the feature descriptor is used to calculate the similarity between key points in the rendered image and the actual image, and is used for subsequent nearest neighbor search, mismatch removal, and geometric registration.

[0036] Through the above steps, the sets of key points in the virtual image and the real image and their corresponding feature descriptors can be obtained respectively, thus providing a foundation for establishing cross-domain feature correspondence and achieving high-precision geometric registration.

[0037] Step S200: Based on the feature descriptors corresponding to the key point locations, perform multi-head feature correlation analysis and adaptive threshold segmentation on the registered virtual image and the real image to obtain an anomaly detection mask. ; It should be noted that step S100 is mainly used to mine the deep semantic consistency of the image, to achieve robust detection of assembly anomaly regions, high-precision spatial alignment between virtual and real images, and reduce appearance deviations caused by viewpoint differences, scale changes and local geometric distortions. On this basis, step S200 further utilizes the semantic correlation between deep features to perform difference characterization on the registered virtual and real images, thereby improving the accuracy and stability of assembly anomaly detection in scenarios such as complex backgrounds, local reflections and unclear textures.

[0038] Specifically, step S200 includes: Step S201: Input the virtual image and the real image into the image encoding network to extract multi-layer deep features; combine the key point positions and their corresponding feature descriptors to perform association constraints on the deep features to enhance the semantic consistency between corresponding regions across domains.

[0039] The image encoding network preferably employs a SAM image encoder, which consists of multiple stacked Transformer layers. Each Transformer layer includes a multi-head self-attention module for extracting global contextual information and local structural information of the image. Keypoint locations are used to provide spatial correspondence constraints, and feature descriptors are used to provide local appearance consistency constraints, thereby improving the reliability of deep feature alignment between virtual and real images.

[0040] Among them, the multi-layer depth features are the SAM image encoder's first... Layer The key feature surfaces corresponding to each attention head are denoted as the key feature vectors of the virtual image. Key feature vectors of real images .

[0041] Step S202: Based on the multi-layer deep features, calculate the comprehensive similarity response map between the virtual image and the real image in the multi-head attention space. To characterize the feature correlation between virtual and real images under different attention heads.

[0042] Among them, the multi-head attention space is a deep feature representation space composed of key feature vectors corresponding to multiple attention heads in each Transformer layer of the SAM image encoder, which is used to model the feature correlation between virtual images and real images under different attention heads.

[0043] Specifically, step S202 includes: Step S2021: Based on the key feature vector of the virtual image Key feature vectors of real images Calculate the corresponding cosine similarity matrix. :

[0044] in, and These represent the virtual image feature location index and the real image feature location index, respectively; For virtual images in the first Layer The first thing to pay attention to. Key feature vectors at each position; For the real image in the first Layer The first thing to pay attention to. Key feature vectors at each position.

[0045] It should be noted that the cosine similarity matrix is ​​used to measure the degree of correspondence between virtual and real images in the deep semantic feature space; through the multi-head attention mechanism, fine-grained associations between virtual and real images can be modeled from different subspaces, thereby realizing multi-head feature correlation analysis.

[0046] Step S2022: Obtain the cosine similarity matrix from multiple attention heads Weighted fusion or linear mapping is performed to obtain a comprehensive similarity response map. , represented as:

[0047] in, To focus on the number of heads, For the first The weighting coefficient of each attention head. This is a linear layer or mapping function used to fuse multi-head similarity results into a single-channel similarity response map.

[0048] Step S203: Based on the comprehensive similarity response map mean Standard deviation Total number of pixels , No. Similarity value at each pixel position Using the formula Calculate the skewness of the similarity distribution ; Among them, the skewness of the similarity distribution Used to describe the asymmetry of pixel value distribution in the comprehensive similarity response map, it can reflect the ratio between normal and abnormal regions in the current scene.

[0049] Step S204: Based on skewness Using the formula Dynamically adjust the binarization threshold ,in, This serves as a baseline threshold to provide the basic segmentation level. This is a sensitivity factor used to control the impact of skewness changes on the threshold adjustment range; This is a correction term used to compensate for system offsets caused by different image scenes, different coding layer feature distributions, or different data acquisition conditions, thereby improving the stability and adaptability of threshold setting.

[0050] Step S205: Combine the similarity response map With binarization threshold Comparison to generate an anomaly detection mask .

[0051] Specifically, when the similarity value at a certain position in the comprehensive similarity response graph is less than the binarization threshold... When the similarity value at a given location is greater than or equal to the binarization threshold, the location is determined to be an abnormal pixel region. When this occurs, the pixel at that location is determined to be a normal region pixel. This yields the anomaly detection mask. Its expression is:

[0052] in, Indicates the first This location belongs to the assembly anomaly area. Indicates the first These locations are within the normal range.

[0053] Through the above steps, based on the completion of geometric registration of virtual and real images, the semantic consistency between virtual and real images can be modeled by combining local description information of key points and deep multi-head attention features. Furthermore, the stable extraction of abnormal regions can be achieved through adaptive threshold segmentation based on skewness, thereby improving the robustness and accuracy of assembly anomaly detection in complex industrial scenarios.

[0054] Step S300: Based on anomaly detection mask Based on the SAM model, abnormal areas are screened from coarse to fine to achieve accurate judgment and contour segmentation of assembly abnormalities.

[0055] Specifically, step S300 includes: Step S301: Input the virtual image into the full-image segmentation mode of the SAM model to generate a set of category-independent candidate part masks. ; Step S302: Calculate the mask for each candidate part With anomaly detection mask overlap rate :

[0056] in, This represents the number of pixels in the intersection region between the candidate part mask and the initial anomalous mask. This represents the total number of pixels in the candidate part mask. A geometric threshold is set. , retain satisfaction The candidate regions are used as candidate anomaly objects.

[0057] Step S303: Perform semantic similarity matching on candidate abnormal objects, and extract the Mask Embedding of the virtual image and the real image within the corresponding mask regions, denoted as... and And calculate the cosine similarity. :

[0058] in, Used to characterize the semantic consistency of candidate regions between virtual and real images.

[0059] Step S304: Combine geometric overlap rate and semantic similarity The system performs decision-making and screening on candidate anomalies, retaining geometrically consistent regions with significant semantic differences as the final anomaly regions, and outputs the anomaly contour segmentation results.

[0060] Specifically, calculate the anomaly determination score of candidate anomaly objects. :

[0061] in, To integrate the weighting coefficients, and ; Used to characterize the geometric consistency between the candidate region and the anomaly detection mask; Used to characterize the semantic consistency of candidate regions between virtual and real images; Used to characterize the degree of semantic difference in candidate regions.

[0062] By setting an anomaly detection threshold , retain satisfaction The candidate regions are selected as the final anomaly regions, and the anomaly contour segmentation results are output.

[0063] Example 2 The constructed dataset is as follows Figure 2 As shown, the data includes a CAD model of the experimental assembly platform, real-world images of the platform taken with an RGB camera, and anomaly masks obtained through manual annotation. The CAD model of the experimental assembly platform covers various complex assembly scenarios, including combinations of parts of different shapes, sizes, and materials, to simulate diverse assembly tasks in real industrial production. The real-world images taken with the RGB camera were acquired under different lighting conditions and shooting angles to fully reflect the complexity and uncertainty of the real assembly environment. The manually annotated anomaly masks detail various possible assembly anomalies, such as missing parts, misassembly, and foreign objects, providing accurate annotation information for model training and evaluation.

[0064] The implementation of this invention includes the following steps ( Figure 1 ): Align virtual and real data. The process is as follows: Figure 3 As shown, after obtaining the feature descriptor sets of the two images, a preliminary feature matching pair is established using a nearest neighbor search strategy. A random sampling consensus algorithm is used for geometric constraint filtering, iteratively calculating and selecting the optimal set of interior points that satisfy the perspective transformation relationship, thereby eliminating outliers. Based on the selected high-quality matching point pairs, the homography matrix describing the transformation from the real image plane to the virtual image plane is calculated. Finally, this transformation matrix is ​​applied to perform inverse perspective transformation and resampling on the real image, correcting it to a pixel coordinate system completely consistent with the virtual image.

[0065] A comparison strategy based on multi-layer features. The process is as follows: Figure 4As shown, the model averages and fuses the similarity matrices calculated from all attention heads in this layer to generate the final global similarity map, Smap. In this map, high similarity values ​​represent normal areas where the virtual and real data are consistent, while low similarity values ​​intuitively point to potential assembly anomalies such as missing parts, misassembly, or foreign object intrusion. When the similarity distribution exhibits a specific skewness, the function automatically adjusts the cutoff point to accurately separate abnormal pixels at the long tail of the distribution.

[0066] Abnormal score fusion and decision-making. The process is as follows: Figure 5 As shown, the confidence score is calculated according to the formula. When the confidence score is lower than the preset threshold, it indicates that the region not only differs in pixels but also undergoes a fundamental change in semantic essence, thus ultimately being determined as an assembly anomaly. Through this cascaded decision-making mechanism of "geometric screening-semantic verification," the anomaly segmentation map output by the model not only eliminates environmental noise but also clearly delineates the specific outline of the abnormal parts based on the high-quality segmentation boundary of SAM.

[0067] The final results were evaluated using precision, recall, F1 score, and ROC-AUC curve as experimental metrics.

[0068] Accuracy:

[0069] Recall rate:

[0070] F1 score:

[0071] ROC-AUC curve: The receiver operating characteristic (ROC) curve describes the relationship between the true positive rate (TPR, or recall) and the false positive rate (FPR). The false positive rate is defined as:

[0072] By continuously adjusting the classification threshold of the model, the (FPR, TPR) coordinate points corresponding to each threshold are calculated, and then all these points are connected to form the ROC curve.

[0073] In the assembly anomaly detection task, the detection results are classified into the following four cases: TruePositive (TP): The pixel or part that is actually an assembly anomaly (discrepancy area) and is correctly predicted as an anomaly by the model; False Positive (FP): The pixel that is actually a normal assembly area (background or correct part) but is incorrectly predicted as an anomaly by the model (i.e., false alarm); True Negative (TN): The pixel that is actually a normal assembly area and is correctly predicted as normal by the model; False Negative (FN): The pixel that is actually an assembly anomaly but is incorrectly predicted as normal by the model (i.e., missed detection).

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall fall within the protection scope of the present invention.

[0075] In summary, this invention discloses an anomaly detection method for complex assembly scenarios based on virtual-real comparison. This method comprises three main steps: virtual-real data alignment, a comparison strategy based on multi-layer features, and anomaly score fusion and decision-making. Specifically, virtual-real data alignment achieves high-precision geometric registration through a SuperPoint network based on self-supervised deep learning; multi-layer feature comparison employs multi-head feature correlation analysis and skewness-based adaptive threshold segmentation; and anomaly decision-making relies on a geometric-semantic dual refinement strategy using the SAM model to complete hierarchical screening and accurate judgment. This invention can effectively improve the robustness and accuracy of anomaly detection in complex assembly scenarios, efficiently eliminate discrete noise interference, and is applicable to various complex equipment assembly scenarios with stringent assembly quality requirements.

[0076] The above description merely illustrates the embodiments of this application, and while it is quite specific and detailed, it should not be construed as limiting the scope of the patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Furthermore, any parts not detailed in this application are conventional techniques.

Claims

1. A method for anomaly detection in complex assembly scenarios based on virtual-real comparison, characterized in that, The methods include: Step S100: Acquire virtual and real images, extract key point positions and feature descriptors to achieve high-precision geometric registration between virtual and real images; Step S200: Based on the feature descriptors corresponding to the key point locations, perform multi-head feature correlation analysis and adaptive threshold segmentation on the registered virtual image and the real image to obtain an anomaly detection mask. ; Step S300: Based on anomaly detection mask Based on the SAM model, abnormal areas are screened from coarse to fine to achieve accurate judgment and contour segmentation of assembly abnormalities.

2. The method according to claim 1, characterized in that, Step S100 includes: Step S101: Collect virtual and real images of the same target object, and use them as inputs for virtual and real images respectively; use the SuperPoint network model based on self-supervised deep learning to preprocess the grayscale values ​​of the virtual and real images to obtain the corresponding single-channel grayscale images; Step S102: Input the virtual image and the real image after grayscale preprocessing into the shared encoder respectively, and extract the shared feature maps corresponding to the two; Step S103: Input the shared feature map into the key point decoder and the feature descriptor decoder. Output the pixel-level key point positions through the key point decoder and output the feature descriptor corresponding to the key point positions through the feature descriptor decoder.

3. The method according to claim 1, characterized in that, Step S200 includes: Step S201: Input the virtual image and the real image into the image encoding network to extract multi-layer deep features; combine the key point positions and their corresponding feature descriptors to perform association constraints on the deep features to enhance the semantic consistency between corresponding regions across domains; Step S202: Based on the multi-layer deep features, calculate the comprehensive similarity response map between the virtual image and the real image in the multi-head attention space. To characterize the feature correlation between virtual and real images under different attention heads; Step S203: Based on the comprehensive similarity response map mean Standard deviation Total number of pixels , No. Similarity value at each pixel position Using the formula Calculate the skewness of the similarity distribution ; Step S204: Based on skewness Using the formula Dynamically adjust the binarization threshold ,in, This serves as a baseline threshold to provide the basic segmentation level. This is a sensitivity factor used to control the impact of skewness changes on the threshold adjustment range; This is a correction term used to compensate for system offsets caused by different image scenes, different coding layer feature distributions, or different data acquisition conditions, thereby improving the stability and adaptability of threshold setting; Step S205: Combine the similarity response map With binarization threshold Comparison to generate an anomaly detection mask .

4. The method according to claim 3, characterized in that, Step S202 includes: Step S2021: Based on the key feature vector of the virtual image Key feature vectors of real images Calculate the corresponding cosine similarity matrix. : in, and These represent the virtual image feature location index and the real image feature location index, respectively; For virtual images in the first Layer The first thing to pay attention to. Key feature vectors at each position; For the real image in the first Layer The first thing to pay attention to. Key feature vectors at each position; Step S2022: Obtain the cosine similarity matrix from multiple attention heads Weighted fusion or linear mapping is performed to obtain a comprehensive similarity response map. , represented as: in, To focus on the number of heads, For the first The weighting coefficient of each attention head. This is a linear layer or mapping function used to fuse multi-head similarity results into a single-channel similarity response map.

5. The method according to claim 3, characterized in that, Step S205 includes: When the similarity value at a certain position in the comprehensive similarity response graph is less than the binarization threshold When the similarity value at a given location is greater than or equal to the binarization threshold, the location is determined to be an abnormal pixel region. When this occurs, the pixel at that location is determined to be a normal region pixel; thus, an anomaly detection mask is obtained. Its expression is: in, Indicates the first This location belongs to the assembly anomaly area. Indicates the first These locations are within the normal range.

6. The method according to claim 1, characterized in that, Step S300 includes: Step S301: Input the virtual image into the full-image segmentation mode of the SAM model to generate a set of category-independent candidate part masks. ; Step S302: Calculate the mask for each candidate part With anomaly detection mask overlap rate : in, This represents the number of pixels in the intersection region between the candidate part mask and the initial anomalous mask. This represents the total number of pixels in the candidate part mask; a geometric threshold is set. , retain satisfaction The candidate regions are used as candidate anomaly objects; Step S303: Perform semantic similarity matching on candidate abnormal objects, and extract the Mask Embedding of the virtual image and the real image within the corresponding mask regions, denoted as... and And calculate the cosine similarity. : in, Used to characterize the semantic consistency of candidate regions between virtual and real images; Step S304: Combine geometric overlap rate and semantic similarity The system performs decision-making and screening on candidate anomalies, retaining geometrically consistent regions with significant semantic differences as the final anomaly regions, and outputs the anomaly contour segmentation results.

7. The method according to claim 6, characterized in that, Step S304 includes: Calculate the anomaly score of candidate anomaly objects : in, To integrate the weighting coefficients, and ; Used to characterize the geometric consistency between the candidate region and the anomaly detection mask; Used to characterize the semantic consistency of candidate regions between virtual and real images; Used to characterize the semantic differences of candidate regions; by setting an anomaly detection threshold. , retain satisfaction The candidate regions are selected as the final anomaly regions, and the anomaly contour segmentation results are output.

8. An anomaly detection device for complex assembly scenarios based on virtual-real comparison, characterized in that, The device is used to implement the anomaly detection method for complex assembly scenarios based on virtual-real comparison as described in claims 1-7.

Images