Unsupervised defect detection method and system driven by frequency domain and fusing multi-modal information

Abstract

The application discloses a kind of by frequency domain driving fusion multimodal information's unsupervised defect detection method and system, the method includes the following steps: multimodal feature acquisition, defect simulation driven by frequency domain, adaptive modal alignment, feature fusion and reconstruction, defect detection and positioning;The application can realize the collaborative optimization of 2D and 3D information driven by frequency domain, realize the accurate alignment of multimodal feature and defect simulation with frequency domain as bridge, and can complete defect detection and positioning without abnormal sample labeling;The application realizes full-type defect coverage by fusing the recognition ability of 2D modal to apparent defects such as color, scratch and printing error, and the recognition ability of 3D modal to structural defects such as deformation and warping;And the application uses the modal alignment strategy driven by frequency domain, only aligns the low-frequency and mid-frequency components of multimodal feature, retains high-frequency component, avoids the loss of modal information caused by global alignment, and fully gives the complementary advantages of 2D and 3D modal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial machine vision detection, in particular to a frequency domain driven fusion multi-modal information unsupervised defect detection. BACKGROUND

[0002] The existing industrial product defect detection technology mainly includes two types:

[0003] One type is based on 2D image analysis. Although this technology has detection capability in some scenarios, it has inherent limitations: the detection result is easily disturbed by light, shadow and object surface optical properties (such as color and texture), resulting in insufficient system robustness, and 2D image as a planar projection of 3D object loses depth information and cannot identify structural defects such as deformation, warping and size out-of-tolerance.

[0004] Another type is based on 3D data anomaly detection. This technology identifies structural defects by analyzing the spatial form of the object and is not sensitive to changes in light. However, traditional 3D detection uses supervised learning and requires a large number of accurately labeled abnormal samples. In actual industrial scenarios, abnormal samples are rare and 3D data labeling costs are extremely high, making it difficult to scale the deployment of supervised 3D detection.

[0005] To solve the data labeling problem, 3D unsupervised anomaly detection technology has gradually developed. It learns the normal sample mode and judges the data deviating from the normal mode as abnormal. However, this technology relies solely on geometric information and has detection blind spots: for samples with no geometric structure difference but with surface defects such as color and printing (such as yellow stains on a pure white plastic shell), pure 3D modal cannot identify and may cause missed detection.

[0006] Although some research attempts to fuse 2D and 3D multi-modal data, existing fusion schemes mostly use global feature alignment, which easily loses modal specific information and cannot fully utilize the complementary advantages of the two modalities, making it difficult to meet the detection needs of complex industrial scenarios. SUMMARY

[0007] The present application aims to provide a frequency domain driven fusion multi-modal information unsupervised defect detection method and system, thereby solving all or one of the above problems in the prior art.

[0008] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0009] On the one hand, the present application provides a frequency domain driven fusion multi-modal information unsupervised defect detection method, comprising the following steps:

[0010] Multi-modal feature acquisition:

[0011] 2D images and 3D depth images of normal samples without defects are collected, and feature extraction is performed on the 2D images and the 3D depth images respectively to obtain 2D neural network features and 3D neural network features;

[0012] Defect simulation in frequency domain:

[0013] The 2D neural network features are mapped to a frequency domain spectrum, and the frequency domain spectrum is divided into frequency bands according to energy distribution, and frequency domain filtering is performed according to the divided frequency bands to obtain high frequency components, medium frequency components and low frequency components; and defect 2D image features are simulated based on perturbed high frequency components, the medium frequency components and the low frequency components;

[0014] Adaptive modal alignment:

[0015] A projection layer is constructed, and the 2D neural network features and the defect 2D image features are projected into a 3D feature space respectively to obtain projected normal 2D features and abnormal 2D features; and the corresponding medium and low frequency components between the normal 2D features and the 3D neural network features are aligned in frequency domain;

[0016] Feature fusion and reconstruction:

[0017] Feature fusion and reconstruction:

[0018] Defect detection and positioning:

[0019] 3D depth images of a sample to be tested are obtained, and 3D features to be tested of the 3D depth images are extracted, and the 3D features to be tested are input into the trained decoder to obtain reconstructed features; and the existence of defects is determined based on the position difference between the reconstructed features and the 3D features to be tested.

[0020] Further, the 2D neural network features are mapped to a frequency domain spectrum, and the frequency domain spectrum is divided into frequency bands according to energy distribution, and frequency domain filtering is performed according to the divided frequency bands to obtain high frequency components, medium frequency components and low frequency components, further comprising:

[0021] The 2D neural network features are subjected to graph Fourier transform, and the corresponding features are mapped to a frequency domain spectrum;

[0022] The frequency domain spectrum is uniformly divided into low frequency, medium frequency and high frequency according to energy distribution;

[0023] The low-frequency component, the medium-frequency component and the high-frequency component are obtained by performing frequency domain filtering on the low-frequency band, the medium-frequency band and the high-frequency band respectively using a frequency domain response function.

[0024] Further, the simulation of the 2D image feature with defects based on the high-frequency component, the medium-frequency component and the low-frequency component further comprises:

[0025] The high-frequency component is subjected to Gaussian noise disturbance to obtain a disturbed high-frequency component.

[0026] The low-frequency component, the medium-frequency component and the disturbed high-frequency component are recombined into a complete frequency domain spectrum, which is mapped back to the original feature space through inverse Fourier transform to obtain a disturbed 2D feature simulating abnormal frequency domain disturbance; the disturbed 2D feature is taken as the 2D image feature with defects.

[0027] Further, the frequency domain alignment of the corresponding medium and low frequency components between the normal 2D feature and the 3D neural network feature further comprises:

[0028] Discrete Fourier transform is performed on the normal 2D feature and the 3D neural network feature respectively to obtain the frequency domain spectrum of the normal 2D feature and the frequency domain spectrum of the 3D neural network feature.

[0029] The first low-frequency component and the first medium-frequency component of the frequency domain spectrum of the normal 2D feature are extracted.

[0030] The second low-frequency component and the second medium-frequency component of the frequency domain spectrum of the 3D neural network feature are extracted.

[0031] The cosine distance is taken as the optimization objective to minimize the distance between the first low-frequency component and the second low-frequency component and the distance between the first medium-frequency component and the second medium-frequency component.

[0032] Further, the feature fusion and splicing based on the abnormal 2D feature, the aligned normal 2D feature and the aligned 3D neural network feature to obtain normal multi-modal fusion features and abnormal multi-modal fusion features further comprises:

[0033] The aligned normal 2D feature and the aligned 3D neural network feature are spliced to obtain the normal multi-modal fusion features.

[0034] The abnormal 2D feature and the aligned 3D neural network feature are spliced to obtain the abnormal multi-modal fusion features which are abnormal in 2D modality and normal in 3D modality.

[0035] Further, the decoder training based on the normal multi-modal fusion feature, the abnormal multi-modal fusion feature, the reconstruction loss and the inverse reconstruction loss further comprises:

[0036] inputting the normal multi-modal fusion feature into a decoder to obtain a normal decoding feature, defining a reconstruction loss, and narrowing the distance between the normal decoding feature and the aligned 3D neural network feature based on the reconstruction loss;

[0037] inputting the abnormal multi-modal fusion feature into the decoder to obtain an abnormal decoding feature, defining an inverse reconstruction loss, and using the inverse reconstruction loss to widen the distance between the abnormal decoding feature and the aligned 3D neural network feature.

[0038] Further, the extraction of the to-be-tested 3D feature of the 3D depth image further comprises:

[0039] The to-be-tested 3D feature of the 3D depth image is extracted by using the same feature extractor as in the multi-modal feature acquisition step.

[0040] Further, the judgment of the defect existence condition based on the position difference between the reconstruction feature and the to-be-tested 3D feature further comprises:

[0041] The cosine distance of each corresponding position in the reconstruction feature and the to-be-tested 3D feature is calculated, and an abnormality map is integrated according to the cosine distance.

[0042] A defect positioning threshold and a defect judgment threshold are set, and the to-be-tested sample is subjected to defect positioning and defect judgment according to the abnormality map, the defect positioning threshold and the defect judgment threshold.

[0043] Further, the defect positioning and defect judgment of the to-be-tested sample according to the abnormality map, the defect positioning threshold and the defect judgment threshold further comprises:

[0044] The cosine distance of the first position in the abnormality map is compared with the defect positioning threshold, and the first position in the to-be-tested sample with the cosine distance greater than the defect positioning threshold is regarded as a defect area.

[0045] The maximum value of the cosine distance of all positions in the abnormality map is compared with the defect judgment threshold, and if the maximum value is greater than the defect judgment threshold, the to-be-tested sample is determined as a defect sample.

[0046] On the other hand, the present application also provides an unsupervised defect detection system driven by frequency domain and fusing multi-modal information, comprising:

[0047] A multi-modal feature acquisition module is configured to: collect 2D images and 3D depth images of normal samples without defects, and perform feature extraction on the 2D images and the 3D depth images respectively to obtain 2D neural network features and 3D neural network features;

[0048] A frequency domain driven defect simulation module is configured to: map the 2D neural network features to a frequency domain spectrum, perform frequency band division on the frequency domain spectrum according to energy distribution, perform frequency domain filtering according to the divided frequency bands, and obtain high-frequency components, medium-frequency components and low-frequency components; simulate 2D image features with defects based on the perturbed high-frequency components, the medium-frequency components and the low-frequency components;

[0049] An adaptive modal alignment module is configured to: construct a projection layer, project the 2D neural network features and the 2D image features with defects into a 3D feature space respectively to obtain projected normal 2D features and abnormal 2D features; and perform frequency domain alignment on corresponding medium-low frequency components between the normal 2D features and the abnormal 2D features.

[0050] A feature fusion and reconstruction module is configured to: perform feature fusion and splicing based on the aligned normal 2D features, the aligned abnormal 2D features, the 2D neural network features and the 3D neural network features to obtain normal multi-modal fusion features and abnormal multi-modal fusion features; and perform decoder training based on the normal multi-modal fusion features, the abnormal multi-modal fusion features, a reconstruction loss and an inverse reconstruction loss.

[0051] A defect detection and positioning module is configured to: acquire a 3D depth image of a sample to be tested, extract a 3D feature to be tested of the 3D depth image, input the 3D feature to be tested into the trained decoder to obtain a reconstructed feature; and determine a defect existence condition based on a position difference between the reconstructed feature and the 3D feature to be tested.

[0052] The technical scheme of the present application has the following advantages:

[0053] 1. The present application can realize frequency domain driven 2D and 3D information collaborative optimization, and the core is to realize accurate alignment and defect simulation of multi-modal features by taking frequency domain as a bridge, without the need for abnormal sample labeling to complete defect detection and positioning.

[0054] 2. The present application solves the problem of missing detection of apparent defects in pure 3D unsupervised detection, and through the fusion of the recognition ability of 2D modal for color, scratches, printing errors and other apparent defects and the recognition ability of 3D modal for deformation, warping and other structural defects, full-type defect coverage is realized, and the technical gap of traditional single-modal detection is filled.

[0055] 3、The application adopts a modal alignment strategy driven by the frequency domain, only aligns the low-frequency and medium-frequency components of the multi-modal features, retains the high-frequency components, avoids the loss of modal information caused by global alignment, and fully gives play to the complementary advantages of 2D and 3D modalities.

[0056] 4、The defect simulation method based on the frequency domain of the application does not need to rely on real abnormal samples, but can generate simulated abnormalities only through frequency domain disturbance of normal 2D features, combines an unsupervised training paradigm, avoids the industry pain points of scarcity of abnormal samples and high labeling cost, and significantly reduces the technical landing threshold.

[0057] 5、The application stage of the application can complete detection only through 3D depth images, does not need to synchronously collect 2D images, adapts to the application demand that part of the 3D data can be obtained in the industrial scene, and realizes accurate positioning and quantitative judgment of defects through cosine distance calculation, the detection precision and robustness are significantly better than those of the prior art, a reliable quality control scheme is provided for complex industrial scenes, and the intelligent detection level of the manufacturing industry is promoted.

[0058] 6、The unsupervised defect detection system driven by the frequency domain and fusing multi-modal information can realize the unsupervised defect detection method driven by the frequency domain and fusing multi-modal information through mutual cooperation of system modules. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0060] Figure 1 is a flowchart of the unsupervised defect detection method driven by the frequency domain and fusing multi-modal information according to the embodiment 1 of the application;

[0061] Figure 2 is a logic diagram of steps S200-S400 in the unsupervised defect detection method driven by the frequency domain and fusing multi-modal information according to the embodiment 1 of the application;

[0062] Figure 3 is a logic diagram of step S500 in the unsupervised defect detection method driven by the frequency domain and fusing multi-modal information according to the embodiment 1 of the application;

[0063] Figure 4 is a logic diagram of step S600 in the unsupervised defect detection method driven by the frequency domain and fusing multi-modal information according to the embodiment 1 of the application;

[0064] Figure 5 is the architecture schematic diagram of the unsupervised defect detection system driven by frequency domain fusion of multi-modal information according to embodiment 2 of the present application. DETAILED DESCRIPTION

[0065] The preferred embodiments of the present application are described in detail below with reference to the accompanying drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the scope of protection of the present application is more clearly defined.

[0066] In the description of the present application, it should be noted that the embodiments described in the present application are part of the embodiments of the present application, not all the embodiments; all other embodiments obtained by those skilled in the art without creative labor on the basis of the embodiments in the present application, belong to the scope of protection of the present application.

[0067] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0068] It should be noted in the description of the present application that: for the optimization of the projection layer, the most naive way is to directly bring the 2D features after projection and the 3D features closer; however, 2D and 3D data have some unique properties of their own, so this completely unconstrained approximation forces the global similarity between the two modal features, thereby weakening the specificity of each modality; in the present application, only the low and medium frequency parts between the modalities are aligned, which mainly depict the main shapes of the object, which are consistent in various modalities; at the same time, the high frequency part of each modality is kept unchanged, because it represents the difference in details of the objects of different modalities.

[0069] Embodiment 1, the present embodiment provides a kind of unsupervised defect detection method driven by frequency domain fusion of multi-modal information, as shown in Figures 1-4 The method comprises the following steps:

[0070] S100, multi-modal data acquisition step, comprising:

[0071] RGB image of 2D defect-free normal sample is synchronously collected by using RGB-D industrial imaging equipment with 3D depth image , set the device parameters consistent, avoid subsequent feature processing due to parameter difference error.

[0072] It should be noted that, since the present application is an unsupervised detection method, only normal sample data needs to be collected, without obtaining abnormal samples, reducing the data acquisition cost.

[0073] S200, cross-modal feature extraction step, comprising:

[0074] ResNet50 network pre-trained in large-scale data set (such as ImageNet data set) is selected as the feature extractor, and the 2D image and the 3D depth image are respectively processed. and Feature extraction is performed to obtain 2D neural network features and 3D neural network features ; wherein H, W and C represent the height, width and channel number of the feature map respectively.

[0075] It should be noted that the parameters of the ResNet50 network are frozen during the training process to ensure that the feature extractor has general feature extraction capability and avoid overfitting to the current detection task data.

[0076] S300, frequency domain driven defect simulation step, comprising:

[0077] S301, frequency domain conversion operation:

[0078] GFT is performed on , the feature is mapped to the frequency spectrum (denoted as ), and the conversion process is realized through the graph Fourier transform operator , and the conversion formula is as follows:

[0079] ;

[0080] wherein is the 2D feature after the projection layer;

[0081] wherein , indicates the orthogonal matrix of the feature vector.

[0082] S302, frequency band division operation:

[0083] According to the energy distribution, the is evenly divided into three frequency bands of low frequency, medium frequency and high frequency;

[0084] The frequency domain response function is used for frequency domain filtering to obtain corresponding low frequency, medium frequency and high frequency components (denoted as , , ), the corresponding filtering process is represented as follows:

[0085] ;

[0086] wherein, , is the eigenvalue;

[0087] wherein, is 1 in the corresponding frequency band and 0 otherwise, and is specifically represented as:

[0088] ;

[0089] In the above formula, and represent the exponents of the low frequency, medium frequency and high frequency band boundaries in the frequency domain spectrum, is used to separate the low frequency and the medium frequency, is used to separate the medium frequency and the high frequency.

[0090] S303, defect simulation operation:

[0091] In order to obtain this special defect which behaves abnormally on the 2D modality and normally on the 3D modality, so that the subsequent training process can identify this kind of defect, the corresponding defect simulation process is carried out, as follows:

[0092] The high frequency component is disturbed by a Gaussian noise with a mean of 0 and a variance of , to obtain the disturbed high frequency component ;

[0093] Recombine , and to form a complete frequency domain spectrum, which is mapped back to the original feature space by inverse graph Fourier transform (IGFT), to obtain the disturbed 2D feature with simulated abnormal frequency domain disturbance , which can be regarded as a 2D image feature containing defects.

[0094] It should be noted that, compared with the frequency domain spectrum of normal samples, the frequency domain spectrum of defect samples usually has stronger high frequency energy, because the low frequency and medium frequency information of the object can reflect the macroscopic characteristics of the object (such as overall shape and overall structure); in contrast, the high frequency highlights the surface texture and detailed structure of the object; therefore, the defect part usually behaves as a high frequency part transformation in the frequency domain spectrum, based on this idea, the present application mainly considers disturbing the high frequency part of the frequency domain to realize the simulation of defects with stable overall shape and prominent defect part.

[0095] S400, adaptive modality alignment step, comprising:

[0096] S401, Projection Layer Construction Operation:

[0097] Since the reconstruction-based unsupervised detection paradigm requires the features input to the decoder to be in similar feature spaces, it is necessary to use a learnable projection layer to project the 2D features, which serve as auxiliary information, into the feature space of the 3D features. Therefore, in this step, a learnable projection layer composed of multiple fully connected layers is set up to project the 2D features, which serve as auxiliary information, into the feature space of the 3D features. and Projecting each feature into a 3D feature space yields the projected normal 2D features. With abnormal 2D features ;

[0098] It should be noted that 2D features The feature representation after the projection layer is as follows: It can be modeled as a collection A graph model with 10 nodes In the formula Represents the set of vertices. Represents the set of edges. The set of weights representing the edges; Each value in Represents vertices and vertex Weights between them; combined Laplace operator It can be decomposed into In the formula Degree matrix, The orthogonal matrix representing the eigenvectors. Includes eigenvalues .

[0099] S402, Frequency Domain Alignment Operation:

[0100] right and Perform discrete Fourier transforms on each to obtain their frequency domain spectra;

[0101] extract Low-frequency components of the frequency domain spectrum Intermediate frequency components ;as well as Low-frequency components of the frequency domain spectrum Intermediate frequency components (The corresponding components can be obtained based on the frequency domain filtering function);

[0102] Using cosine distance as the optimization objective, minimize and , and The distance and the corresponding alignment loss formula are as follows:

[0103] ;

[0104] wherein, denotes the cosine distance calculation.

[0105] It should be noted that in order to achieve effective 2D to 3D modal alignment while maintaining modal specific features, the present application selectively aligns only the low and medium frequency components of the modal.

[0106] S403, abnormal feature processing operation:

[0107] For The above frequency domain alignment operation is not performed, and the abnormal 2D feature after projection is directly used for subsequent fusion to ensure the integrity of the simulated defects.

[0108] S500, feature fusion and reconstruction step, including:

[0109] S501, optimization feature splicing operation:

[0110] Splice and to obtain normal multi-modal fusion features ;

[0111] Splice and to obtain abnormal multi-modal fusion features that are abnormal in 2D modal and normal in 3D modal .

[0112] S502, decoder training operation:

[0113] Input into the decoder to obtain normal decoding features , define reconstruction loss , make and close;

[0114] Input into the same decoder to obtain abnormal decoding features , define inverse reconstruction loss , make and far.

[0115] It should be noted that the unsupervised detection principle based on reconstruction is that if only normal samples are seen during the reconstruction process, the reconstruction process will remember the normal mode without defects, and once an abnormal sample is encountered during testing, the abnormal area cannot be reconstructed, which can be used to judge and locate defects. Therefore, the inverse reconstruction loss is used to help the model to judge which 2D abnormal and 3D normal abnormal conditions, i.e. the abnormal features is successfully reconstructed as .

[0116] S503, total loss optimization operation:

[0117] The total loss of the training process is , and the projection layer and the decoder parameters are optimized by backpropagation until the loss converges.

[0118] S600, defect detection and positioning step:

[0119] S601, sample processing operation:

[0120] Without providing the 2D image of the test sample, the 3D depth image of the test sample is extracted using the same feature extractor in step S200 , and the reconstructed features are obtained by inputting the trained decoder .

[0121] S602, similarity calculation operation:

[0122] Calculate and the cosine distance of the corresponding position to generate an anomaly map , the formula is as follows:

[0123] ;

[0124] Wherein, The larger the value in , the higher the defect probability of the corresponding position.

[0125] S603, defect determination operation:

[0126] Set the defect positioning threshold, filter each position in , and determine the defect area if the position exceeds the threshold;

[0127] Take the maximum value of all values in as the overall defect score of the sample, set the overall determination threshold, and if the score exceeds the threshold, the sample is determined to be defective, otherwise it is normal.

[0128] It should be noted that the above examples are only for the purpose of explaining the present application and cannot limit the protection scope of the present application.

[0129] Embodiment 2, based on the same inventive concept as the unsupervised defect detection method driven by frequency domain fusion of multi-modal information described in embodiment 1, provides an unsupervised defect detection system driven by frequency domain fusion of multi-modal information, as shown in Figure 5 , comprising:

[0130] a multi-modal feature acquisition module, configured to: collect a 2D image and a 3D depth image of a normal sample without defects, and perform feature extraction on the 2D image and the 3D depth image respectively to obtain 2D neural network features and 3D neural network features;

[0131] a frequency domain driven defect simulation module, configured to: map the 2D neural network features to a frequency domain spectrum, perform frequency band division on the frequency domain spectrum according to energy distribution, and perform frequency domain filtering on the divided frequency bands to obtain high frequency components, medium frequency components and low frequency components; and simulate 2D image features with defects based on perturbed high frequency components, the medium frequency components and the low frequency components;

[0132] an adaptive modal alignment module, configured to: construct a projection layer, project the 2D neural network features and the 2D image features with defects into a 3D feature space respectively to obtain projected normal 2D features and abnormal 2D features, and perform frequency domain alignment on corresponding medium and low frequency components between the normal 2D features and the abnormal 2D features;

[0133] a feature fusion and reconstruction module, configured to: perform feature fusion and splicing based on the aligned normal 2D features, the aligned abnormal 2D features and the 3D neural network features to obtain normal multi-modal fusion features and abnormal multi-modal fusion features; and perform decoder training based on the normal multi-modal fusion features, the abnormal multi-modal fusion features, a reconstruction loss and an inverse reconstruction loss;

[0134] a defect detection and positioning module, configured to: acquire a 3D depth image of a sample to be tested, extract 3D features to be tested of the 3D depth image, input the 3D features to be tested into the trained decoder to obtain reconstructed features, and determine whether defects exist based on a position difference between the reconstructed features and the 3D features to be tested.

[0135] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0136] It should be understood that, in various embodiments herein, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments herein.

[0137] It should also be understood that, in the embodiments herein, the term "and / or" only describes an association relationship for associated objects, and can represent three relationships. For example, A and / or B can represent three cases of A existing alone, A and B existing together, and B existing alone. In addition, the character " / " in the present embodiment generally represents an "or" relationship between the front and rear associated objects.

[0138] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present embodiment.

[0139] In several embodiments provided herein, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0140] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments herein.

[0141] In addition, each functional unit in each embodiment herein can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0142] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions herein or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments herein. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0143] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made by using the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. An unsupervised defect detection method driven by frequency domain and fusing multimodal information, characterized in that, Includes the following steps: Multimodal feature acquisition: 2D images and 3D depth images of defect-free normal samples are acquired, and feature extraction is performed on the 2D images and 3D depth images respectively to obtain 2D neural network features and 3D neural network features; Frequency-domain driven defect simulation: The 2D neural network features are mapped to the frequency domain spectrum, and the frequency domain spectrum is divided into frequency bands according to the energy distribution. Frequency domain filtering is performed according to the divided frequency bands to obtain high-frequency components, mid-frequency components and low-frequency components. Based on the perturbation high-frequency component, the mid-frequency component, and the low-frequency component, simulate the defective 2D image features; Adaptive modal alignment: A projection layer is constructed to project the 2D neural network features and the defective 2D image features onto the 3D feature space, respectively, to obtain the projected normal 2D features and abnormal 2D features. Frequency domain alignment is performed on the corresponding low- and mid-frequency components between the normal 2D features and the 3D neural network features; Feature fusion and reconstruction: Based on the abnormal 2D features, the aligned normal 2D features, and the aligned 3D neural network features, feature fusion and splicing are performed to obtain normal multimodal fusion features and abnormal multimodal fusion features; Decoder training is performed based on normal multimodal fusion features, abnormal multimodal fusion features, reconstruction loss, and inverse reconstruction loss; Defect detection and location: Acquire a 3D depth image of the sample to be tested, extract the 3D features to be tested from the 3D depth image, and input the 3D features to be tested into the trained decoder to obtain reconstructed features; The presence of defects is determined based on the positional differences between the reconstructed features and the 3D features to be tested.

2. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The step of mapping the 2D neural network features to a frequency domain spectrum, dividing the frequency domain spectrum into frequency bands according to the energy distribution, and performing frequency domain filtering according to the divided frequency bands to obtain high-frequency components, mid-frequency components, and low-frequency components further includes: Perform a graphical Fourier transform on the features of the 2D neural network to map the corresponding features to the frequency domain spectrum; The frequency domain spectrum is uniformly divided into low frequency, mid frequency, and high frequency according to the energy distribution; Frequency domain response functions are used to perform frequency domain filtering on the divided low-frequency, mid-frequency, and high-frequency bands to obtain the low-frequency component, the mid-frequency component, and the high-frequency component.

3. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The method of simulating defective 2D image features based on perturbation high-frequency components, mid-frequency components, and low-frequency components further includes: Gaussian noise perturbation is applied to the high-frequency components to obtain the perturbed high-frequency components; The low-frequency component, the mid-frequency component, and the perturbed high-frequency component are reconstructed into a complete frequency domain spectrum, and mapped back to the original feature space by inverse graph Fourier transform to obtain the perturbation 2D feature simulating the abnormal frequency domain perturbation; the perturbation 2D feature is then used as the defective 2D image feature.

4. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The step of frequency domain alignment of the corresponding low- and mid-frequency components between the normal 2D features and the 3D neural network features further includes: Perform Discrete Fourier Transform on the normal 2D features and the 3D neural network features respectively to obtain the frequency domain spectrum of the normal 2D features and the frequency domain spectrum of the 3D neural network features; Extract the first low-frequency component and the first mid-frequency component of the frequency domain spectrum of the normal 2D feature; Extract the second low-frequency component and the second mid-frequency component of the frequency domain spectrum of the 3D neural network features; Using cosine distance as the optimization objective, the distance between the first low-frequency component and the second low-frequency component, as well as the distance between the first mid-frequency component and the second mid-frequency component, are minimized.

5. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The step of fusing and concatenating features based on the abnormal 2D features, the aligned normal 2D features, and the aligned 3D neural network features to obtain normal multimodal fusion features and abnormal multimodal fusion features further includes: The aligned normal 2D features are concatenated with the aligned 3D neural network features to obtain the normal multimodal fusion features; The abnormal 2D features are concatenated with the aligned 3D neural network features to obtain the abnormal multimodal fusion features that exhibit abnormal behavior in the 2D modality but normal behavior in the 3D modality.

6. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The decoder training based on normal multimodal fusion features, abnormal multimodal fusion features, reconstruction loss, and inverse reconstruction loss further includes: The normal multimodal fusion features are input into the decoder to obtain normal decoded features. A reconstruction loss is defined, and the distance between the normal decoded features and the aligned 3D neural network features is narrowed based on the reconstruction loss. The abnormal multimodal fusion features are input into the decoder to obtain abnormal decoding features. An inverse reconstruction loss is defined and used to widen the distance between the abnormal decoding features and the aligned 3D neural network features.

7. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The step of extracting the 3D features to be measured from the 3D depth image further includes: The same feature extractor used in the multimodal feature acquisition step is used to extract the 3D features to be measured from the 3D depth image.

8. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 1, characterized in that: The step of determining the presence of defects based on the positional difference between the reconstructed features and the 3D features to be tested further includes: Calculate the cosine distance between the reconstructed feature and each corresponding position in the 3D feature to be tested, and integrate the anomaly map based on the cosine distance; Set a defect location threshold and a defect determination threshold, and perform defect location and defect determination on the sample to be tested based on the anomaly map, the defect location threshold, and the defect determination threshold.

9. The unsupervised defect detection method based on frequency domain-driven fusion of multimodal information as described in claim 8, characterized in that: The step of locating and determining defects in the sample under test based on the anomaly map, the defect location threshold, and the defect determination threshold further includes: The cosine distance at the first position in the anomaly image is compared with the defect location threshold, and the first position in the sample to be tested where the cosine distance is greater than the defect location threshold is designated as the defect region. The maximum value of the cosine distance among all positions in the anomaly graph is compared with the defect determination threshold. If the maximum value is greater than the defect determination threshold, the sample to be tested is determined to be a defective sample.

10. An unsupervised defect detection system driven by frequency domain and fusing multimodal information, characterized in that, include: The multimodal feature acquisition module is used to: acquire 2D images and 3D depth images of defect-free normal samples, extract features from the 2D images and 3D depth images respectively, and obtain 2D neural network features and 3D neural network features; The frequency domain-driven defect simulation module is used to: map the features of the 2D neural network to the frequency domain spectrum, divide the frequency domain spectrum into frequency bands according to the energy distribution, and perform frequency domain filtering according to the divided frequency bands to obtain high-frequency components, mid-frequency components and low-frequency components. Based on the perturbation high-frequency component, the mid-frequency component, and the low-frequency component, simulate the defective 2D image features; An adaptive modal alignment module is used to: construct a projection layer to project the 2D neural network features and the defective 2D image features onto a 3D feature space, respectively, to obtain the projected normal 2D features and abnormal 2D features; Frequency domain alignment is performed on the corresponding low- and mid-frequency components between the normal 2D features and the abnormal 2D features; The feature fusion and reconstruction module is used to: perform feature fusion and splicing based on the aligned normal 2D features, the aligned abnormal 2D features and the 3D neural network features to obtain normal multimodal fusion features and abnormal multimodal fusion features; Decoder training is performed based on normal multimodal fusion features, abnormal multimodal fusion features, reconstruction loss, and inverse reconstruction loss; The defect detection and localization module is used to: acquire a 3D depth image of the sample to be tested, extract the 3D features to be tested from the 3D depth image, and input the 3D features to be tested into the trained decoder to obtain reconstructed features; The presence of defects is determined based on the positional differences between the reconstructed features and the 3D features to be tested.

Images