Unsupervised defect detection method and system based on multi-mode mutual calibration and medium

Abstract

The invention discloses an unsupervised defect detection method and system based on multi-modal mutual calibration, and a medium. The method comprises the following steps: extracting original 2D features and original 3D features of RGB image data and 3D point cloud data; performing mutual projection on the original 2D features and the original 3D features based on a modal projection layer to obtain projected features; performing mutual calibration according to the original 2D features, the original 3D features and the projected features; setting a reconstruction decoder to optimize the mutual calibration process; setting combined training loss to optimize a modal projection layer and a reconstruction decoder; extracting original features of the to-be-detected sample; obtaining projection features of the original features based on the trained modal projection layer; judging the defect condition of the to-be-detected sample according to the difference between the original feature and the projection feature of the to-be-detected sample; according to the method, defect detection can be performed by combining 2D and 3D information without labeling, so that the detection precision is improved, and the application range is widened.

Description

Technical Field

[0001] The present invention relates to the field of industrial machine vision detection technology. Specifically, the present invention is applied to the field of product defect detection, and in particular to an unsupervised defect detection method, system and medium based on multimodal inter-calibration. Background Art

[0002] In industrial production and quality control, defect detection is an important part of ensuring product quality and safety. As the manufacturing industry continues to advance towards automation and intelligence, traditional 2D defect detection methods have gradually exposed their limitations and are unable to meet the inspection needs of complex workpieces and diversified products.

[0003] 2D image information is usually obtained by high-resolution cameras, which can capture details of the surface of objects, such as texture, color, edge and shape features. This information has advantages in detecting surface defects (such as scratches, dents, contamination, etc.). However, due to the lack of depth information, 2D images are often incapable of detecting defects in three-dimensional structures (such as deformation, internal defects, etc.), and are easily affected by factors such as lighting, angle and viewing angle, resulting in reduced detection accuracy. 3D information is usually obtained through lidar, structured light cameras, etc., which can provide spatial structure and depth information of objects and accurately model the three-dimensional form of workpieces. This gives 3D information a unique advantage in detecting structural defects (such as dimensional deviation, geometric deformation, volume damage, etc.). Summary of the invention

[0004] The object of the present invention is to provide an unsupervised defect detection method, system and medium based on multimodal inter-calibration, so as to solve all or one of the above problems existing in the prior art.

[0005] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows: In one aspect, the present invention provides an unsupervised defect detection method based on multimodal inter-calibration, comprising the following steps: Multidimensional feature extraction steps: Collecting RGB image data of a defect-free sample and 3D point cloud data corresponding to the RGB image data; performing deep neural network feature extraction on the RGB image data and the 3D point cloud data respectively to obtain original 2D features and original 3D features; Multimodal information mutual calibration steps: A modal projection layer is set, and the original 2D features and the original 3D features are projected onto each other based on the modal projection layer to obtain projected features; the original 2D features, the original 3D features and the projected features are calibrated onto each other; a reconstruction decoder is set to optimize the mutual calibration process; and a combined training loss is set to optimize the modal projection layer and the reconstruction decoder; Defect detection steps: Acquire a sample to be tested, and extract original features of the sample to be tested based on the multidimensional feature extraction step; acquire projection features of the original features based on the optimized modal projection layer and the reconstruction decoder; and determine the defect condition of the sample to be tested based on the difference between the original features of the sample to be tested and the projection features.

[0006] As an improved solution, the multi-dimensional feature extraction step further includes: Extracting the original 2D features of the RGB image data using a DINO image feature extractor trained based on ImageNet; The original 3D features of the 3D point cloud data are extracted using a PointMAE point cloud feature extractor trained based on ShapeNet.

[0007] As an improved solution, the modal projection layer includes: a first modal projection layer and a second modal projection layer; The post-projection features include: 3D projection features and 2D projection features; The mutually projecting the original 2D features and the original 3D features based on the modal projection layer to obtain projected features includes: Projecting the original 2D features to a 3D modality using the first modality projection layer to obtain the 3D projection features; The original 3D features are projected to the 2D modality using the second modality projection layer to obtain the 2D projection features.

[0008] As an improved solution, the mutual calibration according to the original 2D features, the original 3D features and the projected features includes: Calculating a first cosine distance from the original 2D feature to the 2D projection feature; Calculating a second cosine distance from the original 3D feature to the 3D projection feature; Setting an optimization goal based on minimizing the cosine distance; and setting a first optimization loss for the first modal projection layer and the second modal projection layer according to the optimization goal and the first cosine distance and the second cosine distance; The first modal projection layer and the second modal projection layer are trained and optimized based on the defect-free samples and the first optimization loss, and the mutual calibration is completed based on the training optimization of the first modal projection layer and the second modal projection layer.

[0009] As an improved solution, the reconstruction decoder includes: a first reconstruction decoder and a second reconstruction decoder; The setting of the reconstruction decoder to optimize the mutual calibration process includes: In the mutual calibration process, the first reconstruction decoder and the second reconstruction decoder are used to reconstruct the RGB image data and the 3D point cloud data.

[0010] As an improved solution, the setting of the combined training loss to optimize the modality projection layer and the reconstruction decoder includes: Using the last layer of 2D image features of the RGB image data in the deep neural network feature extraction process, and passing through the first reconstruction decoder to obtain a first reconstructed RGB image; determining a reconstruction loss between the first reconstructed RGB image and the RGB image data as a first training loss; Using the last layer of 3D image features of the 3D point cloud data in the deep neural network feature extraction process, and passing through the second reconstruction decoder to obtain a second reconstructed point cloud; determining a reconstruction loss between the second reconstructed point cloud and the 3D point cloud data as a second training loss; Taking the sum of the first optimization loss, the first training loss and the second training loss as the combined training loss; The modality projection layer and the reconstruction decoder are optimized using the combined training loss.

[0011] As an improved solution, judging the defect of the sample to be tested according to the difference between the original feature of the sample to be tested and the projection feature includes: Based on the Euclidean distance, a pixel-by-pixel feature mapping success comparison is performed between the original features of the sample to be tested and the projection features to obtain a 2D modal anomaly map and a 3D modal anomaly map; Performing a product operation on pixels at corresponding positions in the 2D modal anomaly map and the 3D modal anomaly map to obtain an anomaly detection map; The actual situation of the defect is determined according to the anomaly score of the feature in the anomaly detection graph.

[0012] As an improved solution, the determining the true situation of the defect according to the anomaly score of the feature in the anomaly detection graph further includes: Set anomaly score threshold; In response to the anomaly score of the first defect feature in the anomaly detection image being not lower than the anomaly score threshold, it is determined that the first defect feature actually exists.

[0013] On the other hand, the present invention also provides an unsupervised defect detection system based on multimodal inter-calibration, comprising: A multi-dimensional feature extraction module is used to: collect RGB image data of a defect-free sample and 3D point cloud data corresponding to the RGB image data; perform deep neural network feature extraction on the RGB image data and the 3D point cloud data respectively to obtain original 2D features and original 3D features; A multimodal information mutual calibration module is used to: set a modal projection layer, and project the original 2D features and the original 3D features on each other based on the modal projection layer to obtain projected features; perform mutual calibration according to the original 2D features, the original 3D features and the projected features; set a reconstruction decoder to optimize the mutual calibration process; set a combined training loss to optimize the modal projection layer and the reconstruction decoder; The defect detection module is used to: obtain a sample to be tested, extract the original features of the sample to be tested based on the multidimensional feature extraction step; obtain the projection features of the original features based on the optimized modal projection layer and the reconstruction decoder; and judge the defect condition of the sample to be tested based on the difference between the original features of the sample to be tested and the projection features.

[0014] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the unsupervised defect detection method based on multimodal inter-calibration are implemented.

[0015] On the other hand, the present invention further provides a computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: The memory is used to store computer programs; The processor is used to execute the steps of the unsupervised defect detection method based on multimodal inter-calibration by running the program stored in the memory.

[0016] The beneficial effects of the technical solution of the present invention are: 1. The unsupervised defect detection method based on multimodal mutual calibration described in the present invention can realize defect detection by combining 2D and 3D information without labeling, and can fully utilize the complementary advantages of the two through multimodal fusion technology; in the framework of unsupervised learning, it can realize automatic learning of the normal mode of the object, detect abnormal areas, and realize unsupervised defect detection; through unsupervised multimodal information fusion technology, it makes up for the shortcomings of a single modality, improves the detection accuracy and application scope, and alleviates the problem that some defects in practical applications cannot be detected by a single modality.

[0017] 2. The unsupervised defect detection system based on multimodal inter-calibration described in the present invention can realize the unsupervised defect detection method based on multimodal inter-calibration described in the present invention through the mutual cooperation of system modules.

[0018] 3. The computer-readable storage medium described in the present invention can guide the system modules to cooperate, thereby realizing the unsupervised defect detection method based on multimodal inter-calibration described in the present invention, and the computer-readable storage medium described in the present invention also effectively improves the operability of the unsupervised defect detection method based on multimodal inter-calibration.

[0019] 4. The computer device described in the present invention can store and execute the computer-readable storage medium, thereby realizing the unsupervised defect detection method based on multimodal mutual calibration described in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flowchart of the unsupervised defect detection method based on multimodal inter-calibration according to Embodiment 1 of the present invention; Figure 2 Schematic diagram of the logical architecture of the unsupervised defect detection method based on multimodal inter-calibration according to Embodiment 1 of the present invention; Figure 3 It is a schematic diagram of the architecture of the unsupervised defect detection system based on multimodal inter-calibration described in Example 2 of the present invention. DETAILED DESCRIPTION

[0022] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0023] In the description of the present invention, it should be noted that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0024] The terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0025] Embodiment 1, this embodiment provides an unsupervised defect detection method based on multimodal mutual calibration, such as Figure 1 and Figure 2 As shown, the following steps are included: S100, multi-dimensional feature extraction step: In this step, the original RGB image data of the defect-free normal sample and the 3D point cloud data corresponding to the RGB image data are collected; the deep neural network features are extracted from the RGB image data and the 3D point cloud data respectively, as follows: S101, using RGB images to represent 2D information and using point cloud data to represent 3D information; It should be noted that this method is an unsupervised method, so there is no need to collect defective samples, thereby reducing the additional cost of labeling defective samples.

[0026] S102, using the DINO image feature extractor pre-trained on the large-scale image dataset ImageNet to extract multi-layer deep feature maps of defect-free RGB images; Further, define the RGB image as , define its size as ;in, and Respectively represent the height and width of the RGB image, Indicates the number of channels (for RGB images, is 3); Further, define The features extracted by DINO are: , whose size is The number of channels representing the deep features; Usually less than , in order to obtain The features of each pixel in are transformed into The features of .

[0027] S103, using the PointMAE point cloud feature extractor trained on the large-scale palace dataset ShapeNet to extract multi-layer deep feature maps of the point cloud data; Further, define the 3D point cloud data as , It can be expressed as ;in, is the number of points in the point cloud, Refers to the coordinate values ​​of the three dimensions xyz; Further, define The features extracted by PointMAE are: ; During feature extraction, P The sampling operation is performed on the points, and the The sampling point cloud of ; Since the number of points in the original point cloud data may be very large, point-by-point operation will result in a large amount of calculation, which will cause the feature extraction methods including PointMAE to extract The size is ,and < Therefore, in order to obtain The features of each point in the point cloud are taken into consideration, and the density of the point cloud is improved by using an upsampling method based on neighboring point clouds. Furthermore, the upsampling method based on the neighbor point cloud includes: defining each point in the original point cloud ,Sure In the sampling point cloud space Zhongli The three nearest points , and ;Sure , and Corresponding features , and ;Will , and The weighted average of The eigenvalue of ; The weighted value is , and Respectively to The distance is calculated as follows: in, , and They are arrive The normalized Euclidean distance of .

[0028] S200, multi-modal information mutual calibration step: In this step, two modal projection layers are set, and the original 2D features and the original 3D features extracted in the previous step are projected to each other through the two modal projection layers; the original 2D features and the original 3D features are calibrated to each other according to the mutually projected features; in the above process, two reconstruction decoders are used to prevent over-calibration, as follows: S201, designing a first modal projection layer and a second modal projection layer, using the first modal projection layer to project the original 2D features to the 3D modality to obtain 3D projection features; using the second modal projection layer to project the original 3D features to the 2D modality to obtain 2D projection features; Furthermore, in this embodiment, it is assumed that the collected RGB image and point cloud data are completely aligned, and each pixel in the RGB image has its unique corresponding point cloud point. In actual production, an RGB camera and a depth camera with the same resolution are used to collect images at the same viewing angle, thereby realizing the above assumptions. Furthermore, the first modality projection layer and the second modality projection layer are both learnable mapping layers, and each mapping layer is composed of three 1*1 convolutional layers; among them, the first modality projection layer is represented as , the operation process of the first modal projection layer is expressed by the following formula: ; The second modal projection layer is represented as , the operation process of the second modal projection layer is expressed by the following formula: .

[0029] S202, constraining the similarity between the 2D / 3D projection features and the original 2D / 3D features, thereby achieving mutual calibration between different modalities; Further, calculate the original 2D features To 2D Projection Features The cosine distance is calculated as: ; Calculate the original 3D features Projection features to 3D The cosine similarity distance is calculated as: ; Furthermore, the optimization objective is set to minimize the cosine distance between the feature map calculated based on the two modal input data and the predicted feature map, so the optimization loss of the model training is set to: ; Furthermore, the two mapping layers are trained and optimized based on all defect-free normal samples in the data set and the optimization loss, thereby achieving mutual calibration between different modalities. Furthermore, since DINO and PointMAE are both based on the Transformer structure with the same number of layers, the output features of the shallow encoder in the Transformer contain more detailed local information, and the output features of the deep encoder are more inclined to global semantic generalization; but for the anomaly detection studied by this method, both local information and global generalization are very important, so the above-mentioned mutual calibration of 2D and 3D information should be performed at each layer; but in order to take into account the training efficiency, this method chooses to perform the above calibration at certain layers in the Transformer; for example, when the Transformer has an 8-layer structure, calibration is performed at the 2nd, 5th and 8th layers; Furthermore, some pixels in the 2D image may not actually have corresponding points in the 3D point cloud, such as the background area in the 2D image, whose corresponding point cloud does not exist and is meaningless for the defect detection of this method, so the alignment loss of this part is ignored; It should be noted that in this step, the cosine distance is used to represent the calibration strength from 2D to 3D and from 3D to 2D, so that the two modalities can be well aligned at the deep neural network feature level.

[0030] S203, setting a first reconstruction decoder and a second reconstruction decoder, and in the above-mentioned feature extraction and calibration process, using the first reconstruction decoder and the second reconstruction decoder to reconstruct the original 2D image and 3D point cloud to constrain the modality information itself from excessive deviation, thereby preventing excessive calibration between modalities during the calibration process from causing loss of inherent information of the modality itself; Furthermore, the first reconstruction decoder and the second reconstruction decoder are both composed of three Transformer Blocks, and the Blocks are learnable; the Transformer Block has the same structure as the Transformer Block in the feature extractor Transformer; Furthermore, for the original 2D image, the last layer of 2D image features of the image feature extractor is used The reconstructed RGB image is obtained by the first reconstruction decoder , the training loss of this process is represented as the real original image and The L2 reconstruction loss between ; Furthermore, for 3D point clouds, the last layer of 3D point cloud features of the point cloud feature extractor is used After the second reconstruction decoder, the reconstructed point cloud is obtained , the training loss of this process is expressed as the true original point cloud and The L2 reconstruction loss between ; Further, adopt , and The combination of is used as the training loss of the overall framework of this method , and then optimize the learnable projection layer and feature reconstructor, that is, ; It should be noted that the parameters of DINO and PointMAE are frozen and will not be optimized with the training of other learnable components, so they do not participate in the training process.

[0031] S300, defect detection steps: In this step, the original features (original 2D image features and original 3D point cloud features) of the sample to be tested are extracted according to the same logic as step S200; the original 2D image features and original 3D point cloud features of the sample to be tested are respectively passed through the first modal projection layer and the second modal projection layer to obtain the projection features (3D projection features and 2D projection features) of the sample to be tested; according to the relationship between the original features and the projection features of the sample to be tested, the defect situation of the sample to be tested is judged, as follows: S301, extracting the original features of the sample to be tested, calculating the corresponding projection features, and determining the difference between the original features and the projection features of the sample to be tested; Furthermore, the RGB image and point cloud data of the sample to be tested are subjected to deep feature extraction to obtain features. and ; The feature and After the first modal projection layer and the second modal projection layer After that, we get the features and Features ; Since the defects of the sample to be tested on the 2D level cannot be seen on the 3D level, based on this feature, the feature abnormality can be judged according to the mapping success between the defect 2D feature and the 3D feature, and vice versa; for example, if the sample to be tested has a defect on the 3D level, the defect 3D feature cannot be successfully mapped to the 2D feature during projection, and thus the abnormality is judged; if the sample to be tested has a defect on the 2D level, the defect 2D feature cannot be successfully mapped to the 3D feature during projection, and thus the abnormality is judged (this step is the principle of feature mapping success comparison); S302, calculating a single-mode abnormality judgment graph based on the above difference judgment logic and Euclidean distance; Furthermore, the features obtained above are normalized to improve the computational efficiency of this step. Then, the Euclidean distance is used Extract features (features and ) and projected features (features and Features ) to compare pixel by pixel, and then obtain the abnormal map of the modality and ;in, , .

[0032] S303, performing a pixel product operation on the two modal anomaly images at corresponding pixel positions to obtain a final multimodal fusion anomaly score image; judging whether a corresponding defect exists based on the multimodal fusion anomaly score image; Further, and Perform the product operation of the pixels at corresponding positions to obtain the final anomaly detection map ,and ; Furthermore, since the defect features can be detected in both modes, the defect at this time is an actual defect, so and If the outliers at some locations are very high, then these locations may be defective. In actual operation, the corresponding anomaly score threshold K can be set. When A1≥K somewhere in the equation, it is judged that the defect here is real; It should be noted that this method uses defect-free normal samples to enable the projection layer to learn the cross-modal relationship unique to normal samples; when testing the samples to be tested, due to the differences between normal samples of defective samples, the projection layer trained only on normal samples will have modal projection errors at the defect location; based on this feature, defect detection and positioning can be achieved through the difference between the extracted modal features and the projection features predicted by the cross-modal projection layer.

[0033] It should be noted that the aforementioned samples to be tested include products that are subjected to defect detection during actual operation.

[0034] In summary, this method realizes defect detection by fusing 2D and 3D information, takes advantage of 2D high-resolution surface details and 3D precise spatial structure, automatically learns normal patterns and detects anomalies through unsupervised learning, reduces dependence on labeled data, is suitable for diversified products and changing detection needs, and improves detection accuracy and application scope.

[0035] It should be noted that the above examples are only for explaining the present invention and cannot limit the protection scope of the present invention.

[0036] Embodiment 2, based on the same inventive concept as the unsupervised defect detection method based on multimodal mutual calibration described in Embodiment 1, this embodiment provides an unsupervised defect detection system based on multimodal mutual calibration, such as Figure 3 As shown, including: A multi-dimensional feature extraction module is used to: collect RGB image data of a defect-free sample and 3D point cloud data corresponding to the RGB image data; perform deep neural network feature extraction on the RGB image data and the 3D point cloud data respectively to obtain original 2D features and original 3D features; Specifically, the multi-dimensional feature extraction module includes: a 2D feature extraction unit and a 3D feature extraction unit; A 2D feature extraction unit, configured to extract the original 2D features of the RGB image data using a DINO image feature extractor trained based on ImageNet; The 3D feature extraction unit is used to extract the original 3D features of the 3D point cloud data by using a PointMAE point cloud feature extractor trained based on ShapeNet.

[0037] A multimodal information mutual calibration module is used to: set a modal projection layer, and project the original 2D features and the original 3D features on each other based on the modal projection layer to obtain projected features; perform mutual calibration according to the original 2D features, the original 3D features and the projected features; set a reconstruction decoder to optimize the mutual calibration process; set a combined training loss to optimize the modal projection layer and the reconstruction decoder; Specifically, the modal projection layer includes: a first modal projection layer and a second modal projection layer; Specifically, the post-projection features include: 3D projection features and 2D projection features; Specifically, the multimodal information mutual calibration module includes: a first projection processing unit and a second projection processing unit; A first projection processing unit, configured to project the original 2D feature to a 3D modality using the first modality projection layer to obtain the 3D projection feature; The second projection processing unit is used to project the original 3D features to the 2D modality using the second modality projection layer to obtain the 2D projection features.

[0038] Specifically, the multimodal information mutual calibration module further includes: a cosine distance calculation unit, a first loss calculation unit and a first optimization unit; A cosine distance calculation unit, used to calculate a first cosine distance from the original 2D feature to the 2D projection feature; and calculate a second cosine similarity distance from the original 3D feature to the 3D projection feature; A first loss calculation unit, configured to set an optimization target based on minimizing the cosine distance; and according to the optimization target, setting a first optimization loss for the first modal projection layer and the second modal projection layer according to the first cosine distance and the second cosine distance; The first optimization unit is used to perform training optimization on the first modal projection layer and the second modal projection layer based on the defect-free sample and the first optimization loss, and complete the mutual calibration based on the training optimization of the first modal projection layer and the second modal projection layer.

[0039] Specifically, the reconstruction decoder includes: a first reconstruction decoder and a second reconstruction decoder; Specifically, the multimodal information mutual calibration module further includes: a reconstruction processing unit; A reconstruction processing unit is used to reconstruct the RGB image data and the 3D point cloud data using the first reconstruction decoder and the second reconstruction decoder during the mutual calibration process.

[0040] Specifically, the multimodal information mutual calibration module further includes: a second loss calculation unit and a loss combination optimization unit; A second loss calculation unit is used to use the last layer of 2D image features of the RGB image data in the deep neural network feature extraction process, and obtain a first reconstructed RGB image through the first reconstruction decoder; determine the reconstruction loss between the first reconstructed RGB image and the RGB image data as the first training loss; use the last layer of 3D image features of the 3D point cloud data in the deep neural network feature extraction process, and obtain a second reconstructed point cloud through the second reconstruction decoder; determine the reconstruction loss between the second reconstructed point cloud and the 3D point cloud data as the second training loss; A loss combination optimization unit is used to use the loss sum of the first optimization loss, the first training loss and the second training loss as the combined training loss; and use the combined training loss to optimize the modal projection layer and the reconstruction decoder.

[0041] The defect detection module is used to: obtain a sample to be tested, extract the original features of the sample to be tested based on the multidimensional feature extraction step; obtain the projection features of the original features based on the optimized modal projection layer and the reconstruction decoder; and determine the defect of the sample to be tested according to the difference between the original features of the sample to be tested and the projection features; Specifically, the defect detection module further includes: a feature comparison unit, a pixel product processing unit and a defect judgment unit; A feature comparison unit, used for performing a pixel-by-pixel feature mapping success comparison between the original feature of the sample to be tested and the projection feature based on the Euclidean distance, to obtain a 2D modal anomaly map and a 3D modal anomaly map; A pixel product processing unit, used for performing a product operation on pixels at corresponding positions in the 2D modal anomaly map and the 3D modal anomaly map to obtain an anomaly detection map; A defect judgment unit is used to determine the actual situation of the defect according to the abnormality score of the feature in the abnormality detection image; wherein the defect judgment unit sets an abnormality score threshold; in response to the abnormality score of the first defect feature in the abnormality detection image being not lower than the abnormality score threshold, it is judged that the first defect feature actually exists.

[0042] Embodiment 3: This embodiment provides a computer-readable storage medium, including: The storage medium is used to store computer software instructions used to implement the unsupervised defect detection method based on multimodal inter-calibration described in the above-mentioned Example 1, which includes a program for executing the above-mentioned program set for the unsupervised defect detection method based on multimodal inter-calibration; specifically, the executable program can be built into the unsupervised defect detection system based on multimodal inter-calibration described in Example 2, so that the unsupervised defect detection system based on multimodal inter-calibration can implement the unsupervised defect detection method based on multimodal inter-calibration described in Example 1 by executing the built-in executable program.

[0043] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof.

[0044] Different from the prior art, the present application adopts an unsupervised defect detection method, system and medium based on multimodal mutual calibration, which can combine 2D and 3D information for defect detection without labeling, and through multimodal fusion technology, can fully utilize the complementary advantages of the two; under the framework of unsupervised learning, automatic learning of the normal mode of the object is realized, and abnormal areas are detected to realize unsupervised defect detection; through unsupervised multimodal information fusion technology, the shortcomings of a single modality are compensated, the detection accuracy and application scope are improved, and the problem that some defects in practical applications cannot be detected by a single modality is alleviated.

[0045] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0046] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0047] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0048] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0049] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0050] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0051] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0052] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0053] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An unsupervised defect detection method based on multimodal inter-calibration, characterized in that: The following steps are involved: Multidimensional feature extraction steps: Collecting RGB image data of a defect-free sample and 3D point cloud data corresponding to the RGB image data; performing deep neural network feature extraction on the RGB image data and the 3D point cloud data respectively to obtain original 2D features and original 3D features; Multimodal information mutual calibration steps: A modal projection layer is set, and the original 2D features and the original 3D features are projected onto each other based on the modal projection layer to obtain projected features; the original 2D features, the original 3D features and the projected features are calibrated onto each other; a reconstruction decoder is set to optimize the mutual calibration process; and a combined training loss is set to optimize the modal projection layer and the reconstruction decoder; Defect detection steps: Acquire a sample to be tested, and extract original features of the sample to be tested based on the multidimensional feature extraction step; Acquire projection features of the original features based on the optimized modality projection layer and the reconstruction decoder; The defect condition of the sample to be tested is determined according to the difference between the original feature of the sample to be tested and the projection feature.

2. The unsupervised defect detection method based on multimodal inter-calibration according to claim 1, characterized in that: The multidimensional feature extraction step further comprises: Extracting the original 2D features of the RGB image data using a DINO image feature extractor trained based on ImageNet; The original 3D features of the 3D point cloud data are extracted using a PointMAE point cloud feature extractor trained based on ShapeNet.

3. The unsupervised defect detection method based on multimodal inter-calibration according to claim 1, characterized in that: The modal projection layer comprises: a first modal projection layer and a second modal projection layer; The post-projection features include: 3D projection features and 2D projection features; The mutually projecting the original 2D features and the original 3D features based on the modal projection layer to obtain projected features includes: Projecting the original 2D features to a 3D modality using the first modality projection layer to obtain the 3D projection features; The original 3D features are projected to the 2D modality using the second modality projection layer to obtain the 2D projection features.

4. The unsupervised defect detection method based on multimodal inter-calibration according to claim 3, characterized in that: The mutual calibration according to the original 2D features, the original 3D features and the projected features includes: Calculating a first cosine distance from the original 2D feature to the 2D projection feature; Calculating a second cosine distance from the original 3D feature to the 3D projection feature; Setting an optimization goal based on minimizing the cosine distance; and setting a first optimization loss for the first modal projection layer and the second modal projection layer according to the optimization goal and the first cosine distance and the second cosine distance; The first modal projection layer and the second modal projection layer are trained and optimized based on the defect-free samples and the first optimization loss, and the mutual calibration is completed based on the training optimization of the first modal projection layer and the second modal projection layer.

5. The unsupervised defect detection method based on multimodal inter-calibration according to claim 4, characterized in that: The reconstruction decoder comprises: a first reconstruction decoder and a second reconstruction decoder; The setting of the reconstruction decoder to optimize the mutual calibration process includes: In the mutual calibration process, the first reconstruction decoder and the second reconstruction decoder are used to reconstruct the RGB image data and the 3D point cloud data.

6. The unsupervised defect detection method based on multimodal inter-calibration according to claim 5, characterized in that: The setting of the combined training loss to optimize the modality projection layer and the reconstruction decoder includes: Using the last layer of 2D image features of the RGB image data in the deep neural network feature extraction process, and passing through the first reconstruction decoder to obtain a first reconstructed RGB image; determining a reconstruction loss between the first reconstructed RGB image and the RGB image data as a first training loss; Using the last layer of 3D image features of the 3D point cloud data in the deep neural network feature extraction process, and passing through the second reconstruction decoder to obtain a second reconstructed point cloud; determining a reconstruction loss between the second reconstructed point cloud and the 3D point cloud data as a second training loss; Taking the sum of the first optimization loss, the first training loss and the second training loss as the combined training loss; The modality projection layer and the reconstruction decoder are optimized using the combined training loss.

7. The unsupervised defect detection method based on multimodal inter-calibration according to claim 1, characterized in that: The step of judging the defect of the sample to be tested according to the difference between the original feature of the sample to be tested and the projection feature includes: Based on the Euclidean distance, a pixel-by-pixel feature mapping success comparison is performed between the original features of the sample to be tested and the projection features to obtain a 2D modal anomaly map and a 3D modal anomaly map; Performing a product operation on pixels at corresponding positions in the 2D modal anomaly map and the 3D modal anomaly map to obtain an anomaly detection map; The actual situation of the defect is determined according to the anomaly score of the feature in the anomaly detection graph.

8. The unsupervised defect detection method based on multimodal inter-calibration according to claim 7, characterized in that: The determining the true situation of the defect according to the anomaly score of the feature in the anomaly detection graph further includes: Set anomaly score threshold; In response to the anomaly score of the first defect feature in the anomaly detection image being not lower than the anomaly score threshold, it is determined that the first defect feature actually exists.

9. An unsupervised defect detection system based on multimodal inter-calibration, characterized in that: include: A multi-dimensional feature extraction module is used to: collect RGB image data of a defect-free sample and 3D point cloud data corresponding to the RGB image data; perform deep neural network feature extraction on the RGB image data and the 3D point cloud data respectively to obtain original 2D features and original 3D features; A multimodal information mutual calibration module is used to: set a modal projection layer, and project the original 2D features and the original 3D features on each other based on the modal projection layer to obtain projected features; perform mutual calibration according to the original 2D features, the original 3D features and the projected features; set a reconstruction decoder to optimize the mutual calibration process; set a combined training loss to optimize the modal projection layer and the reconstruction decoder; A defect detection module is used to: obtain a sample to be tested, and extract original features of the sample to be tested based on the multi-dimensional feature extraction step; Acquire projection features of the original features based on the optimized modality projection layer and the reconstruction decoder; The defect condition of the sample to be tested is determined according to the difference between the original feature of the sample to be tested and the projection feature.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the unsupervised defect detection method based on multimodal inter-calibration described in any one of claims 1 to 8 are implemented.

Images