Industrial zero-shot anomaly detection method and system based on cross-modal prompt learning

Abstract

The application discloses an industrial zero-shot anomaly detection method and system based on cross-modal prompt learning, and relates to the technical field of computer vision and deep learning. The method comprises the following steps: acquiring RGB images and point cloud data containing product defects and normal states in an industrial scene, and preprocessing the point cloud data; constructing an anomaly detection model, using the anomaly detection model to preliminarily extract information representation of the RGB images and the point cloud data, and performing further feature extraction and semantic alignment operation through prompt information; training the anomaly detection model based on a cross-modal collaborative mechanism, and testing the trained anomaly detection model by using a collaborative modulation strategy; and using the trained anomaly detection model to perform anomaly detection on products to be detected. The application can significantly improve the real-time defect detection precision and robustness in an industrial environment, especially under the conditions of data scarcity and complex modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to an industrial zero-shot anomaly detection method and system based on cross-modal prompt learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Intelligent manufacturing is a production mode that utilizes advanced information technology and intelligent technology to promote the automation, flexibility and intelligence of the manufacturing process. By integrating cutting-edge technologies such as the Internet of Things, big data, artificial intelligence and cloud computing, intelligent manufacturing can achieve real-time data collection, analysis and intelligent decision-making, significantly improving production efficiency, product quality and enterprise competitiveness. In this process, anomaly detection technology, as one of the key technologies, can help enterprises monitor the production process in real time and timely detect potential defects, thereby ensuring product quality and production safety.

[0004] Anomaly detection refers to the process of identifying unusual or rare objects, events and patterns in images, videos or other data. These anomalies often deviate from the expected pattern or significantly differ from the majority of data. Anomaly detection technology has been widely applied in medical diagnosis, industrial defect detection and video surveillance.

[0005] In industrial production processes, anomaly detection is particularly important. Defects in industrial products, such as bubbles, cracks or uneven coatings, can directly affect product quality and even lead to product scrap. Due to the diversity of defect types and the difficulty of complete labeling, traditional supervised learning methods are limited in their application to industrial defect detection. Therefore, zero-shot anomaly detection technology has gradually become one of the key methods to solve this problem.

[0006] Although zero-shot learning methods can effectively learn features without a large amount of labeled data, they usually rely on the completeness of the data and the consistency of the modalities. Existing zero-shot learning methods lack effective integration of multi-modal information. To address this issue, cross-modal learning techniques that combine RGB images and point cloud data have emerged. Image and point cloud data each provide different perceptual information: RGB images can present the texture and details of objects, while point cloud data contains the three-dimensional geometric structure of objects. By effectively integrating data from these two modalities, the accuracy and robustness of anomaly detection can be significantly improved.

[0007] However, existing technologies still face several challenges, mainly including the following aspects:

[0008] 1. Inter-modal data fusion problem: Although point cloud and RGB image provide their own unique information respectively, how to effectively fuse the two modalities and play a synergistic role in anomaly detection is still a difficulty.

[0009] 2. Zero-shot generalization problem: Although zero-shot learning can reason in the absence of labeled data, how to ensure that the model can accurately make anomaly judgments when facing unknown defects is still a problem to be solved.

[0010] 3. Cross-modal alignment problem: Different modal data have differences in information representation, especially in industrial scenarios, due to the complexity of data and the presence of noise, the problem of modal alignment is more prominent, making it difficult to effectively fuse and contrast modal information.

[0011] In summary, how to realize efficient zero-shot learning of cross-modal data in industrial environments to achieve accurate anomaly detection is a difficult problem that needs to be solved in the prior art. SUMMARY

[0012] In view of the deficiencies of the prior art, the purpose of the present application is to provide an industrial zero-shot anomaly detection method and system based on cross-modal prompt learning, which can significantly improve the real-time defect detection accuracy and robustness in industrial environments, especially under conditions of data scarcity and modal complexity.

[0013] In order to achieve the above purpose, the present application is realized by the following technical scheme:

[0014] The first aspect of the present application provides an industrial zero-shot anomaly detection method based on cross-modal prompt learning, comprising the following steps:

[0015] Obtain RGB images and point cloud data containing product defects and normal states in an industrial scene, and preprocess the point cloud data;

[0016] Construct an anomaly detection model, use the anomaly detection model to preliminarily extract the information representation of the RGB image and the point cloud data, and further perform feature extraction and semantic alignment operations through prompt information;

[0017] Train the anomaly detection model based on a cross-modal collaboration mechanism, and test the trained anomaly detection model using a collaborative modulation strategy;

[0018] Use the trained anomaly detection model to detect anomalies in the product to be detected.

[0019] Further, use a multi-view projection mechanism to perform multi-view rendering preprocessing on the point cloud data, preserve the detailed structure of the point cloud data in two-dimensional space, and generate a two-dimensional depth image suitable for image encoders.

[0020] Further, the information representation of the RGB image is obtained through a frozen image encoder, and the information representation of the point cloud data is generated through a depth image encoding and inverse rendering manner.

[0021] Further, the prompt information is object-independent decoupled semantic prompt information, and the specific steps of creating are as follows:

[0022] The normal prompt and the abnormal prompt are respectively used to represent the normal state and the abnormal state of the image RGB image and the point cloud data;

[0023] The corresponding semantic prompts are designed for the RGB image and the point cloud data;

[0024] The text prompt embedding corresponding to the RGB image and the point cloud data is generated through a pre-trained CLIP text encoder.

[0025] Further, the training process of the anomaly detection model is optimized through constructing an RGB loss function, a point cloud loss function and a cross-modal contrast loss function, wherein the RGB loss function and the point cloud loss function each include a global loss and a local loss.

[0026] Further, the cross-modal contrast loss function realizes the collaborative reinforcement of the RGB image and the point cloud data by aligning the text embeddings of different semantic types.

[0027] Further, the anomaly detection model adopts a dual-modal multi-view comparison architecture to extract the features of the RGB image and the point cloud data, the dual-modal multi-view comparison architecture includes an RGB image feature guide branch and a point cloud feature guide branch, the RGB image feature guide branch is used to extract the texture and color information in the RGB image, and the point cloud feature guide branch focuses on capturing the geometric structure and spatial features of the object.

[0028] The second aspect of the present application provides an industrial zero-shot anomaly detection system based on cross-modal prompt learning, comprising:

[0029] A data acquisition module is configured to acquire RGB images and point cloud data containing product defects and normal states in an industrial scene, and to preprocess the point cloud data;

[0030] A model construction module is configured to construct an anomaly detection model, to preliminarily extract information representation of the RGB image and the point cloud data using the anomaly detection model, and to further extract features and perform semantic alignment operations through prompt information;

[0031] A model training and testing module is configured to train the anomaly detection model based on a cross-modal collaborative mechanism, and to test the trained anomaly detection model using a collaborative modulation strategy;

[0032] An anomaly detection module configured to perform anomaly detection on the product to be detected using the trained anomaly detection model.

[0033] The third aspect of the present application provides a computer readable storage medium storing a computer program, the computer program being adapted to be loaded and executed by a processor to perform the steps of the industrial zero-shot anomaly detection method based on cross-modal prompt learning according to the first aspect of the present application.

[0034] The fourth aspect of the present application provides a computer device, comprising:

[0035] A processor adapted to execute a computer program;

[0036] A computer readable storage medium storing a computer program, the computer program being executed by the processor to implement the industrial zero-shot anomaly detection method based on cross-modal prompt learning according to the first aspect of the present application.

[0037] The above one or more technical solutions have the following beneficial effects:

[0038] The present application provides an industrial zero-shot anomaly detection method and system based on cross-modal prompt learning, which is applied to real-time defect detection in intelligent manufacturing and industrial automation. Existing three-dimensional anomaly detection methods mostly rely on supervised learning and mainly focus on a single modality (such as point cloud), which is difficult to adapt to challenges such as data scarcity and complex modality information in industrial scenarios. The present application innovatively combines RGB images and point cloud data and proposes a zero-shot anomaly detection technology based on cross-modal prompt learning, which significantly improves the accuracy and robustness of anomaly detection. The steps of the present application include: first, collecting RGB images and point cloud data in an industrial scene, and performing multi-view rendering preprocessing on the point cloud to generate a two-dimensional depth image suitable for a visual encoder, ensuring alignment and consistency between different modalities; then, generating unified feature representations of RGB and point cloud through a frozen image encoder and a depth image encoder; then, constructing a cross-modal prompt module to realize independent feature extraction and semantic alignment of the RGB branch and the point cloud branch by introducing object-independent decoupled semantic prompt information; next, designing a cross-modal contrast loss function to improve the feature expression ability of the text modality through positive and negative sample pairs and intra-modal consistency constraints. In the training stage, the cross-modal collaborative mechanism is used to enhance the semantic consistency and feature complementarity between RGB and point cloud, and to improve the anomaly recognition accuracy. In the testing stage, the detection results of the RGB and point cloud branches are dynamically modulated through a collaborative modulation strategy to optimize the complementary reasoning and anomaly judgment of multi-modal information and improve the accuracy of anomaly judgment. Finally, after training and evaluation, the model is deployed to an industrial detection system to realize real-time detection and automatic response of unknown defects using its zero-shot generalization ability.

[0039] The application innovatively combines RGB images and point cloud data, and improves the accuracy and robustness of anomaly detection. The method has better generalization ability, especially in industrial scenes, can detect unknown defects in real time and automatically respond, provides an effective solution for industrial product surface defect detection, and provides strong technical support for quality control and automatic detection in intelligent manufacturing.

[0040] Advantages of additional aspects of the application will be given in part in the following description, some will become apparent from the following description, or will be learned by practicing the application. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0042] Figure 1 Flow chart of the industrial zero-shot anomaly detection method based on cross-modal prompt learning in the embodiment one of the present application;

[0043] Figure 2 Flow chart of the anomaly detection model training stage in the embodiment one of the present application;

[0044] Figure 3 Flow chart of the anomaly detection model test stage in the embodiment one of the present application. DETAILED DESCRIPTION

[0045] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0046] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof;

[0047] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0048] Embodiment one:

[0049] The embodiment one of the present application provides an industrial zero-shot anomaly detection method based on cross-modal prompt learning, as shown in Figure 1 、 Figure 2 and Figure 3 , comprising the following steps:

[0050] S1: Obtain RGB images and point cloud data containing product defects and normal states in an industrial scene, and preprocess the point cloud data.

[0051] In a specific embodiment, the present embodiment uses a multi-view projection mechanism to perform multi-view rendering preprocessing on the point cloud data, preserves the detailed structure of the point cloud data in two-dimensional space, generates a two-dimensional depth image suitable for an image encoder, and ensures the alignment and representation consistency between different modalities.

[0052] Specifically, a structure-preserving multi-view projection mechanism is used to accurately generate a two-dimensional representation of three-dimensional point cloud data. For each point cloud sample , K predetermined viewing angles are set, and a weighted rotation matrix is used to project the three-dimensional point cloud data to a two-dimensional space, wherein the weighted rotation matrix is a combination of rotation matrices along the X and Y axes, which is defined as:

[0053] .

[0054] wherein , denote the rotation matrices around the X and Y axes, , is a dynamically adjusted weighting factor used to control the influence weight of the rotation matrix at each viewing angle. The three-dimensional point cloud data is projected to a two-dimensional depth image by the weighted rotation matrix , that is:

[0055] .

[0056] At the same time, the abnormal label of the point cloud is mapped to the two-dimensional space to obtain a point-level abnormal mask under each viewing angle, and an image-level label corresponding to each viewing angle is generated. , which is calculated as:

[0057] ,

[0058] .

[0059] wherein, and respectively represent the point cloud and the abnormal label mapping function in the projection process, is the point cloud sample.

[0060] The multi-view projection mechanism described in the embodiment can significantly improve the accuracy and detail preservation of point cloud data in two-dimensional space, solve the problems of point cloud sparsity and irregularity in traditional methods, and improve the robustness and accuracy of three-dimensional anomaly detection.

[0061] S2: Construct an anomaly detection model, use the anomaly detection model to preliminarily extract the information representation of the RGB image and the point cloud data, and further perform feature extraction and semantic alignment operations through prompt information.

[0062] S21: Preliminarily extract the information representation of the RGB image and the point cloud data.

[0063] In a specific implementation, the information representation of the RGB image is obtained through a frozen image encoder, and the information representation of the point cloud data is generated through depth image encoding and inverse rendering. The RGB image and the point cloud data form a unified feature representation space.

[0064] Specifically, the information representation of the RGB image is:

[0065] The global feature and the local feature of the RGB image are extracted using a frozen CLIP image encoder, wherein the global feature describes the overall structural information of the image, and the local feature characterizes the detailed information in the image.

[0066] .

[0067] wherein, is the index of the local feature, indicating different local feature levels of the image.

[0068] The information representation of the point cloud data is:

[0069] For the point cloud data , the three-dimensional point cloud data is first converted into multiple two-dimensional depth images ​Each depth image corresponds to a point cloud projected from the k-th view. Then, global features of each depth image are extracted using the frozen CLIP image encoder and local features where the global features capture the geometric structure information, and the local features describe the details of each depth image:

[0070] .

[0071] By processing and averaging the global features of each view , the global feature of the point cloud is generated , which is given by:

[0072] .

[0073] where denotes the splitting of the image of each view, and nv is the number of split parts denotes the stacking of the split features along the view dimension, denotes the averaging of all split blocks to obtain the global feature of the point cloud.

[0074] Then, the local features of each depth image are converted back to the point cloud local features in three-dimensional space , which is given by:

[0075] .

[0076] where denotes the inverse rendering process

[0077] Finally, the representation of the point cloud feature is:

[0078] .

[0079] S22: The anomaly detection model adopts a dual-modal multi-view comparison architecture to extract features of RGB images and point cloud data.

[0080] In one specific embodiment, the dual-modal multi-view comparison architecture includes an RGB image feature guide branch and a point cloud feature guide branch. The RGB image feature guide branch is used to extract texture and color information in the RGB image, and the point cloud feature guide branch focuses on capturing the geometric structure and spatial features of the object. In these two branches, the features of the image and the point cloud are extracted by the respective image encoder, and the corresponding anomaly detection results are generated by comparing with the predefined text prompts.

[0081] Specifically, in the RGB image feature guided branch, first, features are extracted from the input RGB image . Through the pre-trained visual encoder, the global features and multi-level local features of the image are extracted and , and the abnormal detection result map of the RGB image is generated through the comparison and fusion operation with the normal and abnormal text prompts , and the specific calculation formula is:

[0082] .

[0083] wherein, is the weighted matrix of each stage, represents the element-level dot product, is the corresponding text prompt.

[0084] Based on this, the abnormal classification score of the RGB image is calculated by comparing and the text prompt, and the formula is as follows:

[0085] .

[0086] In the point cloud feature guided branch, the input three-dimensional point cloud is rendered into multi-view two-dimensional depth images , and the image global features and local features under each view are extracted through the pre-trained visual encoder. By matching with the corresponding text prompts and , the abnormal detection result map of the point cloud is generated , and the formula is as follows:

[0087] .

[0088] The two-dimensional result of the point cloud is mapped back to the three-dimensional space through the reverse rendering operation to obtain the final point cloud abnormal detection map , and the formula is as follows:

[0089] .

[0090] wherein, is the three-dimensional position information of the point cloud, represents the mapping of two-dimensional information back to the three-dimensional space. Based on this, the point cloud features under multi-view are compared with the text prompt features to calculate the abnormal classification score of the point cloud, and the formula is as follows:

[0091] .

[0092] wherein, represents the number of view angles.

[0093] S23: Further feature extraction and semantic alignment operation is performed through the prompt information.

[0094] In the embodiment, in the process of modal feature extraction and semantic alignment of the RGB image feature guidance branch and the point cloud feature guidance branch, an object-independent decoupled semantic prompt design is proposed to accurately learn the features of RGB images and point cloud data. The prompt information of the design is object-independent decoupled semantic prompt information, which guides the RGB branch and the point cloud branch to perform modal feature extraction and semantic alignment. The specific steps are as follows:

[0095] S231: Create normal prompts and abnormal prompts to represent the normal state and abnormal state of the image RGB image and point cloud data.

[0096] Specifically, by creating normal prompts and abnormal prompts , the normal and abnormal states in the image and point cloud are represented respectively. The normal prompt is mainly used to represent the normal state of the object, and its form is:

[0097] .

[0098] wherein, represents a learnable label related to the normal state of the object, is a label set in the normal state, is a general term representing the object; and the abnormal prompt is used to represent the abnormal state of the object, and its form is:

[0099] .

[0100] wherein, represents a learnable label related to the abnormal state of the object, is a label set in the abnormal state, is used to explicitly identify defects or damages of the object, is a general label of the object.

[0101] S232: Design corresponding semantic prompts for RGB images and point cloud data.

[0102] Specifically, in order to further refine the design of the prompt information, corresponding semantic prompts are created for RGB images and point cloud data. In terms of RGB images, normal RGB prompts and abnormal RGB prompts are designed as:

[0103] ,

[0104] .

[0105] wherein, is the textual label representing the RGB image, and are the labels under normal and abnormal conditions, respectively, represents the generalized label of the object; and in terms of point cloud data, the normal point cloud indicates and the abnormal point cloud indicates is designed as:

[0106] ,

[0107] .

[0108] wherein, is the textual label representing the point cloud data, and are the labels under normal and abnormal conditions, respectively, still represents the generalized label of the object.

[0109] S233: Generate the textual prompt embedding corresponding to the RGB image and the point cloud data pair through the pre-trained CLIP text encoder.

[0110] Finally, all the RGB image and point cloud prompt , , , are generated through the pre-trained CLIP text encoder Corresponding textual prompt embedding: normal RGB prompt embedding , abnormal RGB prompt embedding , normal point cloud prompt embedding and abnormal point cloud prompt embedding .

[0111] S3: Train the anomaly detection model based on the cross-modal collaborative mechanism, and test the trained anomaly detection model using the collaborative modulation strategy.

[0112] S31: Train the anomaly detection model based on the cross-modal collaborative mechanism.

[0113] In one specific implementation, to efficiently perform zero-shot 3D anomaly detection, especially when handling complex anomaly patterns, this embodiment designs a cross-modal collaborative mechanism during the training phase. This mechanism enhances image texture and geometric structure features, strengthens semantic consistency and feature complementarity between modalities, and improves the model's ability to perceive complex anomaly patterns. Furthermore, the training process of the anomaly detection model is optimized by constructing an RGB loss function, a point cloud loss function, and a cross-modal contrastive loss function. Both the RGB and point cloud loss functions include global and local losses, responsible for strengthening the features of their respective modalities. The cross-modal contrastive loss function enhances semantic consistency and feature complementarity between modalities by aligning text embeddings of different semantic types, thereby achieving collaborative enhancement of RGB image and point cloud data.

[0114] S311: Construct a cross-modal contrastive loss function, which enhances the feature representation capability under each modal cue by constructing positive and negative sample pairs and combining them with intramodal consistency constraints.

[0115] Specifically, to address the problem of modal feature learning between RGB images and point cloud data, a cross-modal contrastive loss function is proposed. This function achieves collaborative enhancement of RGB images and point cloud data by aligning text embeddings of different semantic types. Its core idea stems from the hierarchy in visual understanding: point cloud cues capture the overall geometric features of objects, while RGB cues further enhance the understanding of objects through detailed texture information. Therefore, the cross-modal contrastive loss function designed in this embodiment establishes a semantic contrast relationship between point cloud cues and RGB cues, widening the gap between normal and abnormal cues while simultaneously narrowing the gap between cues in the same state, thereby enhancing the discriminative ability in anomaly detection tasks.

[0116] More specifically, the cross-modal contrastive loss function optimizes the cue triples, using point cloud cue embeddings as anchors and adjusting their relationship with RGB cue embeddings. During optimization, the distance between the anchor and positive cues is minimized, while the distance with negative cues is maximized. In this way, the model can effectively learn features of different modalities under both normal and abnormal states and optimize the discriminative ability of each modality. Unified cross-modal contrastive loss. The calculation formula is:

[0117] .

[0118] in, For the first Anchor points for each sample (point cloud hint embedding). and These represent the embedding of positive (normal) and negative (abnormal) hints corresponding to the anchor point, respectively. For Euclidean distance is the minimum distance between the negative sample and the anchor point, is the number of samples.

[0119] When the normal cue of the point cloud is embedded as the anchor point, the normal cue of the RGB image is pulled closer (as a positive example), while the abnormal cue of the RGB image is pushed away (as a negative example); conversely, when the abnormal cue of the point cloud is embedded as the anchor point, the abnormal cue of the RGB image is pulled closer, while the normal cue of the RGB image is pushed away. This double alignment mechanism ensures that the model learns the semantic context between the normal and abnormal cues of the point cloud, so that the model can effectively learn the feature difference between the normal and abnormal, and optimize the discriminative ability of each modality.

[0120] To further enhance the performance of the model, the complementary loss under normal and abnormal conditions is combined to obtain the final total loss function, i.e., the cross-modal contrast loss function :

[0121] .

[0122] wherein the normal-state complementary loss focuses on aligning the cues under normal conditions, and is obtained by embedding the normal point cloud cue as the anchor point, embedding the normal RGB cue as the positive sample, and embedding the abnormal RGB cue as the negative sample; the abnormal-state complementary loss operates under abnormal conditions, and is obtained by embedding the abnormal point cloud cue as the anchor point, embedding the abnormal RGB cue as the positive sample, and embedding the normal RGB cue as the negative sample.

[0123] S312: Construct a point cloud loss function.

[0124] Specifically, in point cloud representation learning, in order to fully integrate three-dimensional semantic information, global and local loss functions are designed respectively. For the local loss, the local features of the depth image of each view of the point cloud are multiplied by the corresponding text cues to obtain the local similarity map from different views, then the local similarity map is mapped from 2D space back to 3D space through inverse rendering operation to obtain the local similarity of the point cloud . In order to accurately model the decision boundary of the abnormal area and alleviate the problem of class imbalance, Dice loss and focal loss are used to obtain the local loss of the point cloud. The calculation formula is as follows:

[0125] .

[0126] wherein, denotes the corresponding ground truth value under each view, denotes the corresponding ground truth value of each point cloud, denotes a series operation, denotes a same size all-1 matrix.

[0127] In global feature learning, the global feature of each depth image under each view is calculated The similarity between the text prompt embedding The global similarity map of the point cloud under different views is obtained , and the global anomaly score of the point cloud is obtained by averaging it . The formula is as follows:

[0128] ,

[0129] .

[0130] In order to calculate the global loss of the point cloud, the cosine similarity (through cross entropy loss) is used to measure the difference between the global features. The calculation formula of the global loss is:

[0131] .

[0132] wherein, denotes the corresponding ground truth value under each view, denotes the ground truth value of each point cloud. The total loss of the point cloud is represented as the sum of the local and global losses:

[0133] .

[0134] S313: Construct an RGB image loss function.

[0135] In terms of RGB image loss learning, the semantic information of the RGB image is also captured from the global and local perspectives. In the local loss, the similarity between the local feature of each RGB image and the text prompt embedding is calculated to generate the local similarity map of the RGB image. The calculation formula of the local loss is as follows:

[0136] .

[0137] For global loss, the similarity between the global feature of each RGB image and the text prompt embedding is calculated to obtain the global similarity map. The calculation formula of the global loss is:

[0138] ​ .

[0139] The total loss of the RGB image is represented as the sum of the local loss and the global loss:

[0140] .

[0141] Finally, the total loss of the CrossMAD framework is composed of the RGB image loss, the point cloud loss, and the cross-modal contrastive loss, which is formulated as follows:

[0142] .

[0143] By optimizing this comprehensive loss function, the model can simultaneously learn the features of RGB images and point cloud data during training, fully leveraging the complementary advantages of both, thereby significantly improving the performance of zero-shot three-dimensional anomaly detection, especially when facing complex anomaly patterns.

[0144] S32: Test the trained anomaly detection model using a collaborative modulation strategy.

[0145] In a specific embodiment, a collaborative modulation strategy is used in the testing phase to dynamically modulate the detection results based on the semantic confidence of each modality branch, achieving complementary reasoning of multi-modal information and final anomaly judgment.

[0146] Due to the domain differences between datasets and the unknown nature of the target, this embodiment proposes a collaborative modulation strategy to explicitly balance the output results of the RGB image and point cloud branches through a hyperparameter This strategy can adapt to the domain differences between training data and target datasets and solve the unknown problem of the detection target during testing, thereby improving the efficiency of the detection process and ensuring the accuracy of the detection results. The specific fusion process can be represented as:

[0147] ,

[0148] .

[0149] wherein, represents a Gaussian filter, controls the proportional weight of the output results of the RGB image and point cloud branches. This weighting strategy can dynamically adjust the contribution of different modalities based on the semantic confidence of each modality, thereby optimizing the final anomaly detection results.

[0150] S4: Use the trained anomaly detection model to perform anomaly detection on the product to be detected.

[0151] In a specific embodiment, the trained model is deployed in an industrial detection system, utilizing its generalization capability under zero-shot conditions to achieve real-time detection and intelligent response of unknown defects.

[0152] During the training process of the anomaly detection model, 15 rounds of training were performed to optimize the RGB cues and point cloud cues. After each round of training, the model was evaluated using an independent test set, and the model weight with the best performance was retained. This strategy ensures that at the end of each round of training, the model with the best performance is selected for further training and optimization, and finally the model with the best performance on the test set is selected for deployment. Through this optimization process, it is ensured that the resulting anomaly detection system can achieve optimal performance in practical applications, thereby improving the accuracy and stability of the system in industrial environments.

[0153] Embodiment two:

[0154] Embodiment two of the present application provides an industrial zero-shot anomaly detection system based on cross-modal cue learning, comprising:

[0155] The data acquisition module is configured to acquire RGB images and point cloud data containing product defects and normal states in an industrial scene, and to preprocess the point cloud data;

[0156] The model construction module is configured to construct an anomaly detection model, use the anomaly detection model to preliminarily extract information representation of the RGB images and the point cloud data, and perform further feature extraction and semantic alignment operation through the cue information;

[0157] The model training and testing module is configured to train the anomaly detection model based on a cross-modal collaborative mechanism, and test the trained anomaly detection model using a cooperative modulation strategy;

[0158] The anomaly detection module is configured to use the trained anomaly detection model to perform anomaly detection on the product to be detected.

[0159] Embodiment three:

[0160] Embodiment three of the present application provides a computer readable storage medium, which stores a computer program, the computer program is suitable for being loaded and executed by a processor to perform the steps in the industrial zero-shot anomaly detection method based on cross-modal cue learning as described in embodiment one of the present application.

[0161] Embodiment four:

[0162] Embodiment four of the present application provides a computer device, comprising:

[0163] The processor is adapted to execute the computer program;

[0164] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps in the industrial zero-shot anomaly detection method based on cross-modal prompt learning according to the embodiment one.

[0165] The steps involved in the above embodiments two, three and four correspond to the method embodiment one, and the specific implementation can refer to the related description part of the embodiment one.

[0166] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0167] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. Computer instructions can be stored in a computer readable storage medium or transmitted by a computer readable storage medium. Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that the computer can access or a data processing device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD) or semiconductor media (for example, solid state disk (SSD)) and the like.

[0168] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for zero-shot anomaly detection in industrial applications based on cross-modal cue learning, characterized in that, Includes the following steps: Acquire RGB images and point cloud data containing product defects and normal states in industrial scenarios, and preprocess the point cloud data; An anomaly detection model is constructed, and the information representation of RGB images and point cloud data is initially extracted using the anomaly detection model. Further feature extraction and semantic alignment operations are then performed using prompt information. An anomaly detection model is trained based on a cross-modal collaborative mechanism, and the trained anomaly detection model is tested using a collaborative modulation strategy. The trained anomaly detection model is used to detect anomalies in the product to be inspected. The training process of the anomaly detection model is optimized by constructing RGB loss function, point cloud loss function and cross-modal contrast loss function, where both RGB loss function and point cloud loss function include global loss and local loss; The cross-modal contrastive loss function optimizes the cue triples, using point cloud cue embeddings as anchors and adjusting their relationship with RGB cue embeddings. During optimization, the distance between the anchor and positive cues is minimized, while the distance with negative cues is maximized. A unified cross-modal contrastive loss is applied. The calculation formula is: ， in, For the first Anchor points for each sample and These represent the embedding of positive and negative hints corresponding to the anchor point, respectively. For Euclidean distance The defined minimum interval controls the minimum distance between negative samples and anchor points. The number of samples; To further enhance model performance, complementary losses under normal and abnormal conditions are combined to obtain the final total loss function, namely the cross-modal contrastive loss function. : ， Among them, normal state complementary loss Focusing on aligning cues under normal conditions involves embedding normal point cloud cues. As an anchor point, a normal RGB hint is embedded. As a positive sample, abnormal RGB hints are embedded. Obtained as a negative sample; anomalous complementary loss To perform operations under abnormal conditions, abnormal point cloud prompts are embedded. As an anchor point, an abnormal RGB indication is embedded. As a positive sample, normal RGB hint embedding It was obtained as a negative sample.

2. The industrial zero-shot anomaly detection method based on cross-modal cue learning as described in claim 1, characterized in that, A multi-view projection mechanism is used to perform multi-view rendering preprocessing on point cloud data, preserving the detailed structure of the point cloud data in two-dimensional space, and generating a two-dimensional depth image suitable for image encoders.

3. The industrial zero-shot anomaly detection method based on cross-modal cueing learning as described in claim 1, characterized in that, Information representation of RGB images is obtained through a frozen image encoder, while information representation of point cloud data is generated through depth image encoding and inverse rendering.

4. The industrial zero-shot anomaly detection method based on cross-modal cue learning as described in claim 1, characterized in that, The prompt message is a decoupling semantic prompt message unrelated to the object. The specific steps for creating it are as follows: Normal and abnormal prompts are created to represent the normal and abnormal states of RGB images and point cloud data, respectively. Design corresponding semantic cues for RGB images and point cloud data; Text prompt embeddings corresponding to RGB images and point cloud data are generated using a pre-trained CLIP text encoder.

5. The industrial zero-shot anomaly detection method based on cross-modal cue learning as described in claim 4, characterized in that, The cross-modal contrastive loss function achieves synergistic enhancement of RGB images and point cloud data by aligning text embeddings of different semantic types.

6. The industrial zero-shot anomaly detection method based on cross-modal cue learning as described in claim 1, characterized in that, The anomaly detection model employs a dual-modal, multi-view comparison architecture to extract features from RGB images and point cloud data. This architecture includes an RGB image feature-guided branch and a point cloud feature-guided branch. The RGB image feature-guided branch is used to extract texture and color information from RGB images, while the point cloud feature-guided branch focuses on capturing the geometric structure and spatial features of objects.

7. An industrial zero-shot anomaly detection system based on cross-modal cue learning, characterized in that, include: The data acquisition module is configured to acquire RGB images and point cloud data containing product defects and normal states in an industrial scene, and to preprocess the point cloud data. The model building module is configured to build an anomaly detection model, use the anomaly detection model to initially extract information representations from RGB images and point cloud data, and perform further feature extraction and semantic alignment operations through prompt information. The model training and testing module is configured to train the anomaly detection model based on a cross-modal collaborative mechanism and to test the trained anomaly detection model using a collaborative modulation strategy. The anomaly detection module is configured to use a trained anomaly detection model to perform anomaly detection on the product to be inspected. The training process of the anomaly detection model is optimized by constructing RGB loss function, point cloud loss function and cross-modal contrast loss function, where both RGB loss function and point cloud loss function include global loss and local loss; The cross-modal contrastive loss function optimizes the cue triples, using point cloud cue embeddings as anchors and adjusting their relationship with RGB cue embeddings. During optimization, the distance between the anchor and positive cues is minimized, while the distance with negative cues is maximized. A unified cross-modal contrastive loss is applied. The calculation formula is: ， in, For the first Anchor points for each sample and These represent the embedding of positive and negative hints corresponding to the anchor point, respectively. For Euclidean distance The defined minimum interval controls the minimum distance between negative samples and anchor points. The number of samples; To further enhance model performance, complementary losses under normal and abnormal conditions are combined to obtain the final total loss function, namely the cross-modal contrastive loss function. : ， Among them, normal state complementary loss Focusing on aligning cues under normal conditions involves embedding normal point cloud cues. As an anchor point, a normal RGB hint is embedded. As a positive sample, abnormal RGB hints are embedded. Obtained as a negative sample; anomalous complementary loss To perform operations under abnormal conditions, abnormal point cloud prompts are embedded. As an anchor point, an abnormal RGB indication is embedded. As a positive sample, normal RGB hint embedding It was obtained as a negative sample.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-6: Industrial Zero-Sample Anomaly Detection Method Based on Cross-Modal Cueing Learning.

9. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the industrial zero-shot anomaly detection method based on cross-modal cueing learning as described in any one of claims 1-6.

Images