A Category-Aware Multimodal Industrial Anomaly Detection and Segmentation Method

Abstract

This invention discloses a category-aware multimodal industrial anomaly detection and segmentation method, belonging to the field of image anomaly detection and segmentation technology. The method includes: extracting multi-level visual features using a visual encoder with an integrated hierarchical enhancement module to construct learnable text prompts containing both normal and anomalous semantics; achieving interactive updating of visual and textual features through a bidirectional cross-attention mechanism, enabling consistent expression of the two modalities in a shared embedding space; generating anomaly maps for a baseline branch and a category-aware branch respectively during the detection phase, and performing inter-layer consistency constraints at the pixel level to ensure spatial and semantic coordination of cross-scale features; and integrating the results of the two branches through adaptive weighted fusion to output the final anomaly detection map. This method utilizes the cross-modal semantic alignment capability of visual language models to achieve anomaly detection and segmentation of industrial images, requiring no anomaly samples, exhibiting high accuracy and strong generalization, and is applicable to various industrial defect detection and segmentation tasks.

Description

Technical Field

[0001] This invention relates to the field of image anomaly detection and segmentation technology, and in particular to a multimodal industrial anomaly detection and segmentation method based on category awareness. Background Technology

[0002] In the field of image anomaly detection and segmentation, especially in industrial production scenarios, challenges such as limited sample size, diverse defect types, and high annotation costs are frequently encountered. Traditional anomaly detection methods typically rely on large amounts of labeled data, making it difficult to adapt to the constantly changing product types and process conditions in industrial settings. Furthermore, due to data privacy and acquisition limitations, obtaining sufficient defect samples for model training is often quite challenging.

[0003] With the development of visual language models, detection methods based on visual-text alignment have gradually emerged. Representative models such as CLIP provide new insights into anomaly detection and segmentation by establishing cross-modal semantic associations between images and text descriptions. However, these models still face significant challenges when applied to industrial inspection. On the one hand, their feature learning process tends to focus on global semantic alignment, neglecting the detailed structure of local regions, resulting in insufficient ability to identify fine-grained anomalies. On the other hand, existing methods often rely on manually written or task-specific text prompts, limiting their adaptability to different product categories or complex defect types. Furthermore, while some improved methods introduce learnable prompts or contextual modeling mechanisms, they still struggle to simultaneously consider category semantic differences and anomaly morphological variations. The lack of effective association constraints between features at different levels and modalities affects the robustness and generalization performance in complex industrial scenarios.

[0004] Therefore, existing industrial anomaly detection methods based on deep learning or visual language models still suffer from insufficient ability to identify anomaly details, limited model generalization, and strong reliance on manual prompts. How to improve the model's ability to perceive complex semantics and local anomalies without requiring a large number of labeled samples, and to achieve efficient alignment between features of different modalities, has become a pressing technical problem in the field of industrial anomaly detection. Summary of the Invention

[0005] The problem to be solved by this invention is to provide a category-aware multimodal industrial anomaly detection and segmentation method. By introducing category-level visual features into text prompts, it can simultaneously model semantic differences between classes and anomaly changes within classes. Combined with hierarchical enhancement and bidirectional cross-attention mechanism, it significantly improves the ability to detect and segment abnormal regions in complex industrial scenarios.

[0006] This invention adopts the following technical solution: a multimodal industrial anomaly detection and segmentation method based on category awareness, comprising the following steps:

[0007] S1. Acquire the input industrial image to be detected, perform preprocessing, unify the image size, and extract visual features of multiple intermediate layers through a visual encoder with an integrated hierarchical enhancement module.

[0008] S2. Based on the predefined normal and abnormal semantic prompt templates, construct text features corresponding to the visual features in S1;

[0009] S3. Input the visual features obtained in S1 and the text features obtained in S2 into the cross-learning network. Through the bidirectional cross-attention mechanism, project the features of the two modalities into the shared embedding space, perform fine-grained semantic alignment of visual and text features, and output cross-modal embedding features.

[0010] S4. Based on the cross-modal embedding features output by S3, the anomaly maps from multiple intermediate layers are upsampled to a uniform scale and fused to generate preliminary anomaly detection results for the baseline branch.

[0011] S5. Input the industrial image to be tested into the category-aware prompting module, extract category-level semantic information, and explicitly embed it into the text prompt template; then, through the processing flow from S1 to S4, perform cross-modal alignment and fusion of multimodal features to generate category-aware branch anomaly detection results.

[0012] S6. The baseline branch preliminary anomaly detection map obtained in S4 and the category-aware branch anomaly detection map obtained in S5 are weighted and fused to generate the final anomaly detection result map.

[0013] As a further improvement of the present invention, step S1 specifically includes:

[0014] S11. Input industrial images to be detected, including normal industrial product images without label information and abnormal industrial product images with label information.

[0015] S12. The size unification operation in the preprocessing step uses preset parameters to unify all input samples to a specified size.

[0016] S13. The visual encoder with integrated hierarchical enhancement module is constructed and improved based on the pre-trained CLIP visual encoder. The feature extraction network includes, but is not limited to, ViT-B-16-224, ViT-L-14-224, ViT-L-14-336, etc., and the visual features are obtained by aligning the outputs of multiple intermediate layers.

[0017] S14. The hierarchical enhancement module constrains the visual features at different levels in the visual backbone network to prevent abrupt changes or semantic shifts in features during inter-layer transmission.

[0018] As a further improvement of the present invention, step S2 specifically includes:

[0019] S21. The normal and abnormal semantic prompt templates are used to guide the features of normal and abnormal semantic states in the visual language model, respectively. They also contain a set of learnable vectors embedded in the text space to learn the unified text context of product categories.

[0020] S22. To achieve effective alignment of cross-modal features, the text embedding space has been structurally improved based on the original CLIP text encoder.

[0021] In the improved CLIP text encoder, in addition to containing normal and abnormal baseline cues, the input sequence of each layer also introduces a continuous learnable embedding at the beginning of the sequence for dynamically modeling hierarchical semantic information. The input sequence after this expansion constitutes the cue sequence of that layer. The cue sequence of each layer is aligned with the multi-scale features of the corresponding layer of the visual encoder, thereby achieving cross-layer semantic consistency and fine-grained feature fusion.

[0022] As a further improvement of the present invention, step S3 specifically includes:

[0023] S31. The cross-learning network projects the visual features obtained in S1 and the text embedding space obtained in S2 onto a unified latent feature space, and achieves fine-grained alignment across levels and modalities by executing a bidirectional cross-attention mechanism.

[0024] S32. In the bidirectional cross-attention mechanism, the positive interactive alignment path uses visual features as guiding information to update the text embedding. This path effectively injects semantic information at the visual level into the text features, thereby obtaining text representations adapted to different levels of visual features.

[0025] S33. The reverse interaction alignment path in the bidirectional cross-attention mechanism is controlled by an adjustable confidence factor. When the semantic alignment between text and vision is insufficient, the reverse path is activated to guide the updating of visual features by the information embedded in the text, thereby strengthening the semantic consistency between weakly aligned modalities.

[0026] As a further improvement of the present invention, step S4 specifically includes:

[0027] S41. By upsampling, the output of the anomaly map in the intermediate layer is adjusted to the same spatial resolution as the input image.

[0028] S42. Multiple intermediate-level anomaly maps of the same size are enhanced through a hierarchical feature enhancement mechanism and fused into a final anomaly detection map, thereby enhancing the spatial and semantic consistency of anomaly maps at different scales.

[0029] Specifically, the anomaly maps of the shallowest (coarse-grained) and deepest (fine-grained) layers are extracted separately, aligned at the pixel level, and then... Consistency constraints based on the loss function can be represented by the following loss function.

[0030] As a further improvement of the present invention, step S5 specifically includes:

[0031] S51. The category-aware prompting module is used to extract the category-level visual features of the image and inject them into the text prompt to achieve category-aware guidance.

[0032] First, the input image is processed by a pre-trained CLIP image encoder to extract global visual features. Then it is input into a lightweight classifier. In this system, the classifier consists of a one-dimensional convolutional layer, a feature transformation layer, and a classification head.

[0033] Simultaneously, the intermediate features at the input of the classification head are selected as the image category representation, denoted as... .

[0034] In the process of constructing text prompts, a learnable gating factor is introduced. It fuses image category-level features with learnable text prompts.

[0035] S52. After obtaining category-aware cues, the same processing flow as S1 to S4 is used to perform cross-modal alignment and fusion.

[0036] Specifically, the input image first passes through a visual encoder that integrates a hierarchical enhancement module to extract visual features from multiple intermediate layers.

[0037] Subsequently, the obtained visual features and the obtained category-aware text features are input into the cross-learning network. Through a bidirectional cross-attention mechanism, the features of the two modalities are projected into a shared embedding space to achieve fine-grained semantic alignment of visual and text features.

[0038] Next, based on the cross-modal embedding features of the output, the anomaly maps from multiple intermediate layers are upsampled to a uniform scale and fused to generate anomaly detection results for the category-aware branch.

[0039] As a further improvement of the present invention, step S6 specifically includes:

[0040] S61. Use a hierarchical weighted fusion strategy in the baseline branch and the category-aware branch respectively to fuse feature information from different semantic levels.

[0041] S62. To integrate the detection results of different branches, the final anomaly detection map is obtained by weighted summation of the output results of the baseline branch and the category-aware branch.

[0042] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0043] This invention presents a multimodal industrial anomaly detection and segmentation method. By introducing a category-aware cueing mechanism and a hierarchical enhancement strategy, it achieves deep fusion and fine-grained semantic alignment of visual and textual modalities. First, a visual encoder with an integrated hierarchical enhancement module extracts multi-level visual features and constructs learnable textual cueing containing both normal and anomalous semantics. Then, a bidirectional cross-attention mechanism enables interactive updating of visual and textual features, ensuring consistent representation of the two modalities within a shared embedding space. During the detection phase, the model generates anomaly maps for both a baseline branch and a category-aware branch, and performs inter-layer consistency constraints at the pixel level to ensure spatial and semantic coordination of cross-scale features. Finally, an adaptive weighted fusion strategy integrates the results from both branches, outputting the final anomaly detection map as feedback. This method efficiently models category differences and anomaly changes without requiring a large number of anomaly samples, significantly improving the localization and segmentation performance of anomaly regions in industrial scenarios. Attached Figure Description

[0044] Figure 1 This is a flowchart of the steps of the multimodal industrial anomaly detection and segmentation method of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating the implementation method of the multimodal industrial anomaly detection and segmentation model according to an embodiment of the present invention;

[0046] Figure 3 These are different types of abnormal images and their abnormal detection maps implemented in embodiments of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0048] In one embodiment of the present invention, a category-aware multimodal industrial anomaly detection and segmentation method is provided. By embedding category-level visual information into prompt templates, the method effectively models the semantic differences between different product categories and the feature differences between normal and abnormal regions. At the same time, by combining hierarchical enhancement and multi-level cross-modal attention mechanisms, a multi-layered interaction between visual and textual features is achieved, thereby improving the detection and segmentation capabilities of subtle abnormal regions. Even under conditions where no product is seen or only a limited number of known categories are known, the abnormal regions can be accurately located.

[0049] Specifically, this embodiment uses a category-aware multimodal industrial anomaly detection and segmentation method, the process of which is as follows: Figure 1 and Figure 2 As shown, the steps are as follows:

[0050] S1. Acquire the input industrial image to be detected and perform preprocessing to unify the image size; extract visual features of multiple intermediate layers through a visual encoder with an integrated hierarchical enhancement module.

[0051] In this embodiment, step S1 involves the acquisition and preprocessing of the industrial image to be detected, as well as multi-layer feature extraction based on an improved visual encoder.

[0052] Specifically, the first step is to acquire an industrial image to be inspected, which may include normal industrial product images without label information and abnormal product images with label information.

[0053] To ensure consistency of input images, a uniform size preprocessing operation is performed on all input images. This preprocessing scales the input images to a specified size using preset parameters, for example, 518 pixels. Bilinear interpolation is then used to adjust the images into square images with width and height of 518 pixels, thus ensuring spatial scale consistency between different images.

[0054] Subsequently, a visual encoder with an integrated hierarchical enhancement module is used to extract features from the preprocessed image. The visual encoder is built based on a pre-trained CLIP visual encoder and improved to adapt to industrial anomaly detection tasks. The pre-trained feature extraction network can adopt architectures including but not limited to ViT-B-16-224, ViT-L-14-224, and ViT-L-14-336.

[0055] During feature extraction, multiple intermediate layer outputs of the encoder are selected as visual features. For example, for an encoder with 24 Transformer layers, the outputs of layers 6, 12, 18 and 24 can be selected evenly as intermediate features to capture multi-scale information from coarse to fine granular.

[0056] To enhance the structural consistency among multi-layer features, this embodiment introduces a hierarchical enhancement module in the visual encoder to perform regularization constraints on visual features at different levels, preventing feature mutations or semantic shifts during inter-layer transmission.

[0057] Specifically, let the extracted multi-layer features be... Mean squared error constraints are applied between features in adjacent layers, and the features are smoothed by the following inter-layer regularization loss:

[0058] ;

[0059] in, This represents the mean square error. The total number of feature layers participating in feature extraction. Indicates the first step through the visual encoder Image features extracted from layers.

[0060] During training, the loss feedback mechanism can further suppress excessive feature shifts, thereby ensuring the spatial and semantic continuity of multi-layer visual features.

[0061] S2. Based on the predefined normal and abnormal semantic prompt templates, construct text features corresponding to the visual features in S1.

[0062] In this embodiment, step S2 involves constructing text features corresponding to visual features based on a predefined semantic template, so as to achieve semantic alignment between visual and text modalities.

[0063] Specifically, firstly, semantic cue templates for representing normal and abnormal states are defined in the text space to guide the model in learning representational features under different semantic conditions. These templates not only contain fixed linguistic description structures but also embed a set of learnable word vectors to capture semantic contextual information for different product categories.

[0064] Normal semantic prompts are defined as: ;

[0065] Anomaly semantic hints are defined as follows: ;

[0066] in, and ( These are the learnable normal and abnormal semantic embedding vectors, belonging to... The Vickers training parameter space is used to model the differences in text features under different semantic scenarios. Location is used to represent the current semantic state. Words such as perfect, normal, and good can be used to describe the normal state, while words such as flawed, abnormal, and damaged can be used to represent the abnormal state. For representing fixed text templates, phrases like "A photo of a" or "There is a photo of" can be used. This structural design allows the model to explicitly distinguish between normal and abnormal semantic attributes at the prompt level.

[0067] To further enhance cross-modal alignment capabilities, this embodiment improves the input structure based on the original CLIP text encoder.

[0068] In the improved text encoder, in addition to the semantic cues mentioned above, the input sequence of each layer also inserts continuous learnable embeddings at the beginning of the sequence to dynamically model hierarchical semantic information, and adds zero vectors at the end of the sequence to maintain consistent input length.

[0069] After expansion, the first The layer prompt sequence can be represented as:

[0070] ;

[0071] ;

[0072] in, and These represent the start and end marker embeddings of the text sequence, respectively, with the latter corresponding to visual features in cross-modal alignment; and These correspond to the embedded representations of the phrases "a photo of a perfect" and "a photo of aflawed," respectively. It is a dynamically learnable embedding used to capture hierarchical semantic changes; This represents a zero-padding matrix used to preserve a fixed input length of 77 tokens; The dimension representing the embedding of normal and abnormal words; and ( ) respectively represent the first Layer normal and abnormal semantic embedding vectors.

[0073] S3. Input the visual features obtained in S1 and the text features obtained in S2 into the cross-learning network. Through the bidirectional cross-attention mechanism, the features of the two modalities are projected into the shared embedding space to achieve fine-grained semantic alignment of visual and text features.

[0074] In this embodiment, step S3 aims to achieve deep semantic fusion and cross-modal alignment of visual and textual features. Specifically, the cross-learning network first performs hierarchical processing on the multi-layer features from the visual encoder to capture visual semantics at different scales; simultaneously, it embeds the normal and abnormal semantic vectors output by the text encoder. Combined, forming a text embedding sequence , This represents the dimension of the latent feature space.

[0075] For visual modalities, the first The image patch features extracted by the layer are represented as follows: , They represent the first The height, width, and length dimensions of the layer patch features.

[0076] Subsequently, the two are jointly projected onto a dimension of In the latent feature space, so as to perform cross-modal attention computation in a unified embedding space.

[0077] The bidirectional cross-attention mechanism consists of a forward semantic injection path and a reverse semantic feedback path.

[0078] In the positive interaction path, visual features are regarded as guiding modalities. Through attention mechanisms, semantic information at the visual level is injected into the text embedding, thereby obtaining a text representation that better fits the visual semantic distribution.

[0079] The calculation process is as follows:

[0080] ;

[0081] in, The query vector represents the text modality. and The first Key-value representation of layered visual features; , , It is a learnable projection matrix used to map the features of each modality to a shared feature space.

[0082] To further improve consistency between weakly semantically aligned modalities, a confidence adjustment mechanism is introduced during the attention computation process. When the cross-modal alignment confidence... When the value falls below a set threshold (e.g., 0.9), the reverse interaction path is initiated.

[0083] At this point, text embedding will serve as guiding information, in turn guiding the updating and reconstruction of visual features, thereby compensating for the shortcomings of unidirectional alignment. Its calculation definition is as follows:

[0084] ;

[0085] in, The query vector representing the visual modality. and These are the key and value vectors of the text modality, respectively. , , It is a learnable projection matrix, which functions similarly to a forward interaction path.

[0086] S4. Based on the cross-modal embedding features output by S3, the anomaly maps from multiple intermediate layers are upsampled to a uniform scale and fused to generate preliminary anomaly detection results for the baseline branch.

[0087] In this embodiment, step S4 aims to generate preliminary anomaly detection results with hierarchical consistency based on the cross-modal embedding features output in step S3. To achieve this goal, anomaly maps from different intermediate layers are upsampled to a uniform scale and then fused to achieve unified representation in terms of spatial resolution and semantic hierarchy.

[0088] First, feature interaction is performed on the two sets of QKV triples obtained in step S3 to generate intermediate anomaly maps for each layer. The calculation process is as follows:

[0089] ;

[0090] in, This indicates a cross-modal attention operation. This indicates an upsampling operation, used to adjust the anomaly maps output from each layer to the same spatial resolution as the input image, resulting in... .

[0091] Subsequently, to further ensure semantic consistency between different levels, a hierarchical feature enhancement mechanism was introduced to perform pixel-level alignment constraints on the anomaly maps of the shallowest (coarse-grained) and deepest (fine-grained) layers, thereby suppressing discontinuities between feature layers.

[0092] The constraint loss is defined as follows:

[0093] ;

[0094] in, The model represents the first time. The anomaly graph generated by the layer, This is the mean squared error function, used to measure the differences between anomaly maps at different levels.

[0095] S5. Input the image to be tested into the category-aware prompt module, extract category-level semantic information, and explicitly embed this information into the text prompt template; then, using the same processing flow as S1 to S4, perform cross-modal alignment and fusion on the enhanced multimodal features to generate the anomaly detection results of the category-aware branch.

[0096] In this embodiment, step S5 is used to construct a category-aware branch and generate category-enhanced anomaly detection results.

[0097] Specifically, the image to be tested is first input into a pre-trained CLIP image encoder to extract global visual representations. The global representation is then fed into a lightweight classifier. The classifier includes a one-dimensional convolutional layer (i.e., a kernel size of 1), several feature transformation layers, and a classification head. In this classifier, the intermediate features at the input of the classification head are taken as the category-level representation of the image, denoted as... ,in, This indicates the number of categories indicated.

[0098] To achieve category injection for text prompts, a learnable gating factor is introduced. The image category features are weighted and fused with pre-defined learnable text embeddings to construct a category-aware cue embedding:

[0099] ;

[0100] in, and Learnable embeddings for normal and abnormal baseline cues, respectively. and This provides category-aware cues for the fused data. Gating factor. It can be learned as a trainable parameter in upstream classifier training or overall system joint training, or it can be set to different initial values ​​in different deployment scenarios to adjust the injection intensity of category information.

[0101] After obtaining category-aware cues, the same cross-modal processing flow as steps S1 to S4 is further adopted: the input image is processed by a visual encoder with an integrated hierarchical enhancement module to extract multi-layer visual features; the visual features are embedded together with the category-aware text constructed in step S5 and input into a cross-learning network, and fine-grained semantic alignment in the shared latent space is achieved through a bidirectional cross-attention mechanism; then, hierarchical consistency enhancement and upsampling are performed to obtain anomaly response maps of each layer, and finally, anomaly detection maps of the category-aware branch are output.

[0102] S6. The image to be tested is processed through the process from S1 to S5. The baseline branch detection map obtained in S4 and the category-aware branch detection map obtained in S5 are weighted and fused to generate the final anomaly detection result map.

[0103] In this embodiment, step S6 is used to fuse the anomaly detection results of the baseline branch and the category-aware branch to generate the final anomaly detection map, such as... Figure 3 As shown. Input image reference. Figure 3 (a) to (d) in the diagram; the corresponding final anomaly detection diagram is referenced. Figure 3 (e)~(h) in the middle.

[0104] Specifically, the multi-layer anomaly maps generated by the baseline branch and the category-aware branch are first subjected to hierarchical weighted fusion. In the baseline branch, let the first... The anomaly graph generated by the layer is The corresponding anomaly graph in the category-aware branch is: The respective fusion results and It can be represented as:

[0105] ;

[0106] ;

[0107] in, For learnable hierarchical weights, satisfying This is used to balance the contributions of features at different semantic levels to the final result.

[0108] Subsequently, the baseline branch and the category-aware branch are fused to obtain the final anomaly detection map. :

[0109] ;

[0110] in, The adjustment coefficient is used to control the relative weights of the baseline branch results and the category-aware branch in the final prediction.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal industrial anomaly detection and segmentation method based on category awareness, characterized in that, Includes the following steps: S1. Acquire the input industrial image to be detected, perform preprocessing, unify the image size, and extract visual features of multiple intermediate layers through a visual encoder with an integrated hierarchical enhancement module. S2. Based on the predefined normal and abnormal semantic prompt templates, construct text features corresponding to the visual features in S1; S3. Input the visual features obtained in S1 and the text features obtained in S2 into the cross-learning network. Through the bidirectional cross-attention mechanism, project the features of the two modalities into the shared embedding space, perform fine-grained semantic alignment of visual and text features, and output cross-modal embedding features. S4. Based on the cross-modal embedding features output by S3, the anomaly maps from multiple intermediate layers are upsampled to a uniform scale and fused to generate preliminary anomaly detection results for the baseline branch. S5. Input the industrial image to be tested into the category-aware prompting module, extract category-level semantic information, and explicitly embed it into the text prompt template; then, through the processing flow from S1 to S4, perform cross-modal alignment and fusion of multimodal features to generate category-aware branch anomaly detection results. Specifically, it includes the following sub-steps: S51. Extract image category-level visual features through the category-aware prompting module and inject text prompts for category-aware guidance: The input image is processed by a pre-trained CLIP image encoder to extract global visual features. , This indicates the fixed token length output by the CLIP image encoder; Global visual features Input to lightweight classifier In the classifier It includes a one-dimensional convolutional layer, a feature transformation layer, and a classification head. It also selects intermediate features from the input of the classification head as image category features, denoted as: , The dimension representing the category hint feature; Text prompts are constructed by introducing learnable gating factors. This involves fusing image category-level features with learnable text prompts; S52. After obtaining category-aware cues, perform cross-modal alignment and fusion using the same processing flow as S1 to S4: The input image is processed by a visual encoder that integrates a hierarchical enhancement module to extract visual features from multiple intermediate layers. The obtained visual features and category-aware text features are input into a cross-learning network. Through a bidirectional cross-attention mechanism, the two modal features are projected into a shared embedding space to perform fine-grained semantic alignment of visual and text features. Based on the cross-modal embedding features of the output, the anomaly maps from multiple intermediate layers are upsampled to a uniform scale and fused to generate anomaly detection results of the category-aware branch; S6. The baseline branch preliminary anomaly detection map obtained in S4 and the category-aware branch anomaly detection map obtained in S5 are weighted and fused to generate the final anomaly detection result map.

2. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, In step S1, the input industrial images to be detected include normal industrial product images without label information and abnormal industrial product images with label information; based on preset parameters, bilinear interpolation is used to scale all input image samples to a specified image size.

3. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, In step S1, the visual encoder is based on the pre-trained CLIP visual encoder and integrates a hierarchical enhancement module. The feature extraction network includes, but is not limited to: ViT-B-16-224, ViT-L-14-224, and ViT-L-14-336. By selecting multiple intermediate layers, image features are extracted evenly, and the aligned output is used as visual features. The hierarchical enhancement module is used to constrain visual features at different levels in the visual backbone network. The multi-layer features extracted by the visual encoder are... The mean squared error constraint is applied between adjacent layers to constrain the differences between features in adjacent layers, and excessive feature shifts are suppressed through loss feedback during training. ； in, This represents the mean square error. The total number of feature layers participating in feature extraction. Indicates the first step through the visual encoder Image features extracted from layers.

4. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, Step S2 includes the following sub-steps: S21. Construct normal and abnormal semantic prompt templates to guide the features of normal and abnormal semantic states in the visual language model, respectively. They also contain a set of learnable vectors embedded in the text space to learn the unified text context of product categories. Normal baseline indication With abnormal baseline indication The structures are defined as follows: ； ； in, and These represent the learnable word embedding vectors for normal and abnormal semantics, respectively. Each embedding vector is Trainable parameters in a dimensional representation space; state words Represents a normal or abnormal semantic state; Indicates a fixed text template; S22. Construct a text embedding space, improve the structure based on CLIP text encoder, expand the input sequence so that the input sequence of each layer contains normal and abnormal baseline prompts, and introduce continuous learnable embeddings at the beginning of the sequence. The expanded input sequence constitutes the cue sequence for each layer, which is aligned with the multi-scale features of the corresponding layer of the visual encoder to achieve cross-layer semantic consistency and fine-grained feature fusion.

5. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, Step S3 includes the following sub-steps: S31. Input the visual features obtained from each layer in S1 and the text embedding space obtained in S2 into the cross-learning network, project them onto a unified latent feature space, and perform fine-grained alignment across layers and modalities by executing a bidirectional cross-attention mechanism. S32. In the bidirectional cross-attention mechanism, the positive interactive alignment path uses visual features as guiding information to inject text features to update the text embedding and obtain text representations that adapt to different levels of visual features. S33. The reverse interaction alignment path in the bidirectional cross-attention mechanism is controlled by an adjustable confidence factor. When the semantic alignment between text and vision is insufficient, the reverse path is activated. The information embedded in the text guides the update of visual features in reverse, thereby strengthening the semantic consistency between weakly aligned modalities.

6. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, Step S4 includes the following sub-steps: S41. By upsampling, the output of the anomaly map in the intermediate layer is adjusted to a spatial resolution consistent with the size of the input image. S42. Multiple intermediate-layer anomaly maps of the same size are enhanced through a hierarchical feature enhancement mechanism. The shallowest coarse-grained anomaly map and the deepest fine-grained anomaly map are extracted separately, aligned at the pixel level, and fused into the final anomaly detection map, thereby enhancing the spatial and semantic consistency of anomaly maps at different scales.

7. The multimodal industrial anomaly detection and segmentation method according to claim 6, characterized in that, In step S42, the consistency constraint is represented by the following loss function: ； in, Indicates the current branch number The anomaly graph generated by the layer, This represents the mean square error.

8. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, In step S51, the fusion process is represented as follows: ； ； in, and Learnable embeddings for normal baseline cues and abnormal baseline cues, respectively. and Embed the fused category-aware cue. Indicates the gating factor. The category-aware prompt module is shown in section 1. Image category features output by the layer.

9. The multimodal industrial anomaly detection and segmentation method according to claim 1, characterized in that, Step S6 includes the following sub-steps: S61. A hierarchical weighted fusion strategy is used in the baseline branch and the category-aware branch respectively to fuse feature information from different semantic levels. The fusion result is as follows: ； ； in, and For the first Baseline branches and category-aware branch anomaly maps generated by the layers. For learnable fusion weights, satisfying ; S62. Based on the detection results of different branches, calculate the output results of the baseline branch and the category-aware branch. and Weighted summation yields the final anomaly detection map: ； in, This is an adjustment coefficient used to control the relative weight of the two branch results in the final detection.

Images