A structural appearance anomaly cross-modal identification method and system based on a diffusion model

Abstract

The application provides a structure appearance anomaly cross-modal identification method and system based on a diffusion model, relates to the technical field of anomaly identification, and comprises the following steps: obtaining a to-be-identified appearance image of a target structure; constructing an anomaly identification instruction for a to-be-identified anomaly type; inputting the to-be-identified appearance image and the anomaly identification instruction into a trained image-text cross-modal identification model to perform a visual task of anomaly identification and obtain a target image containing a semantic segmentation result; wherein the image-text cross-modal identification model is constructed based on a diffusion model, and the visual task of anomaly identification is converted into image generation which is constrained by an image condition and an instruction condition; and the appearance image of an engineering and geological structure and a task instruction are input into the constructed image-text cross-modal identification model to obtain a semantic segmentation result of the image, so that appearance anomaly pixel-level fine identification is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly recognition, and in particular to a structural appearance anomaly cross-modal recognition method and system based on a diffusion model. BACKGROUND

[0002] Engineering and geological structures (including infrastructure engineering, rock and geological structures, etc.) are continuously influenced by environmental factors and load effects during long-term service or natural evolution, and abnormal modes appear. For engineering structures, it is manifested as concrete cracking, steel corrosion, joint leakage, and other material degradation; for geological structures, it may be manifested as rock mass fracture expansion, soil layer sliding, and dissolution cavity, etc. Geological hazards. If these abnormalities are not identified and intervened in time in the early stage of development, local damage often spreads gradually through a chain effect - engineering structures may experience load degradation or sudden collapse, and geological structures are prone to trigger chain disasters such as landslides and ground subsidence. This evolution process from micro-damage to macro-failure eventually significantly threatens the safety of human life and property and the safety of the ecological environment.

[0003] With the rapid development of artificial intelligence and digital image processing technology, intelligent detection based on deep learning has become an important trend in the development of structural appearance anomaly intelligence and automation. The current mainstream deep learning-based anomaly detection methods are mostly based on deep convolutional neural networks, encoder-decoder structures such as U-Net, DeepLabV3+, SegFormer, etc. Deep learning methods show strong advantages compared to traditional methods, but their development is still restricted by two major bottlenecks: first, the model generalization ability is insufficient, and the detection performance significantly decreases when facing complex and variable actual scenarios; second, excessive dependence on a large number of fine-labeled data leads to high model costs and difficulty in quickly adapting to new scenarios.

[0004] Therefore, the existing structural appearance anomaly recognition technology has deficiencies in high precision and strong generalization ability. SUMMARY

[0005] To solve the above problems, the present application proposes a structural appearance anomaly cross-modal recognition method and system based on a diffusion model, which inputs the appearance images and task instructions of engineering and geological structures into the constructed image-text cross-modal recognition model to obtain the semantic segmentation results of the images, and realizes fine-grained pixel-level recognition of appearance abnormalities.

[0006] According to some embodiments, the present application adopts the following technical solutions:

[0007] A structural appearance anomaly cross-modal recognition method based on a diffusion model, comprising:

[0008] obtaining a to-be-recognized appearance image of a target structure;

[0009] construct an abnormality recognition instruction for the abnormality type to be recognized;

[0010] input the appearance image to be recognized and the abnormality recognition instruction into the trained image-text cross-modal recognition model, execute the visual task of abnormality recognition, and obtain a target image containing a semantic segmentation result;

[0011] The image-text cross-modal recognition model is constructed based on a diffusion model, and the visual task of abnormality recognition is converted into image generation constrained by an image condition and an instruction condition.

[0012] According to some embodiments, the present application adopts the following technical scheme:

[0013] A structure appearance abnormality cross-modal recognition system based on a diffusion model comprises:

[0014] An image acquisition module configured to acquire an appearance image to be recognized of a target structure;

[0015] An instruction construction module configured to construct an abnormality recognition instruction for the abnormality type to be recognized;

[0016] An abnormality recognition module configured to input the appearance image to be recognized and the abnormality recognition instruction into the trained image-text cross-modal recognition model, execute the visual task of abnormality recognition, and obtain a target image containing a semantic segmentation result;

[0017] The image-text cross-modal recognition model is constructed based on a diffusion model, and the visual task of abnormality recognition is converted into image generation constrained by an image condition and an instruction condition.

[0018] According to some embodiments, the present application adopts the following technical scheme:

[0019] A computer program product comprising a computer program, which, when executed by a processor, implements the structure appearance abnormality cross-modal recognition method based on a diffusion model.

[0020] According to some embodiments, the present application adopts the following technical scheme:

[0021] A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implements the structure appearance abnormality cross-modal recognition method based on a diffusion model.

[0022] According to some embodiments, the present application adopts the following technical scheme:

[0023] An electronic device comprises a processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the diffusion model-based structural appearance anomaly cross-modal identification method.

[0024] Compared with the prior art, the beneficial effects of the present application are:

[0025] The diffusion model-based structural appearance anomaly cross-modal identification method provided by the present application can deeply understand the general visual features and semantic concepts of various anomalies by introducing the powerful generation prior of the diffusion model, especially by introducing the image guidance coefficient and the instruction guidance coefficient, so as to ensure the balance between semantic analysis and image details, realize the robust and accurate identification of various structural anomalies (such as cracks, peeling, rust, etc.), and greatly improve the generalization and adaptability of the method in complex real environments. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the present application and are incorporated in and constitute a part of this specification. The embodiments of these drawings are set forth to explain the present application and are not intended to limit the present application unduly.

[0027] Figure 1 The method flowchart of example 1. DETAILED DESCRIPTION

[0028] The present application will be further described below with reference to the accompanying drawings and examples.

[0029] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0030] It should be noted that the terms used herein are merely intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should be understood that when the term "comprising" and / or "comprises" is used in the specification, it means that the features, steps, operations, devices, components and / or combinations thereof are present.

[0031] Example 1

[0032] In an embodiment of the present application, a diffusion model-based structural appearance anomaly cross-modal identification method is provided, as shown in Figure 1 The method comprises the following steps:

[0033] Step S1: obtaining a to-be-identified appearance image of a target structure;

[0034] Step S2: constructing an anomaly identification instruction for the type of anomaly to be identified;

[0035] Step S3: inputting the appearance image to be identified and the anomaly identification instruction into the trained image-text cross-modal identification model, performing the visual task of anomaly identification, and obtaining a target image containing a semantic segmentation result;

[0036] The image-text cross-modal identification model is constructed based on a diffusion model, and the visual task of anomaly identification is converted into image generation constrained by image conditions and instruction conditions.

[0037] In recent years, as a new generation of generative model, diffusion models have gradually attracted widespread attention in the academic community due to their superior performance in image generation tasks. Diffusion models recover the original data from noise by simulating the gradual noise process of images, can generate images with higher fidelity, have more flexible model architecture and accurate log-likelihood calculation. At the same time, natural language processing technology has developed rapidly, especially instruction learning and cross-modal modeling technology such as CLIP, which makes it possible to bridge the semantics between images and text, providing a technical basis for "language-guided image editing".

[0038] Based on the above technical trends, the present embodiment proposes a structural appearance anomaly cross-modal identification method based on a diffusion model. Cross-modal refers to the process of information transmission and understanding between different modalities (such as vision, text, etc.). The image-text cross-modal identification model InstructDiffusion based on diffusion model is introduced into the field of infrastructure appearance anomaly detection. Through the construction of a closed-loop framework of "natural language instruction-cross-modal reasoning-pixel-level segmentation", the unified identification and accurate positioning of multiple types of structural anomalies under zero-shot conditions are realized. The present embodiment effectively makes up for the shortcomings of existing methods in terms of generalization and interactivity, and provides a new solution with stronger universality for infrastructure structure intelligent inspection systems. From the perspective of constructing and training the identification model InstructDiffusion, the specific implementation process is described in detail as follows:

[0039] I. Data acquisition and preprocessing

[0040] The appearance anomaly images of the engineering or geological structure are collected by mobile phones, cameras, or drones, etc. The collected images are subjected to uniform image size normalization processing and illumination correction to reduce the interference of external environment on the identification accuracy of the model. At the same time, through image enhancement methods such as edge enhancement and contrast enhancement, the features of target regions such as cracks and corrosion are made more prominent, which facilitates the accurate identification of the subsequent identification model InstructDiffusion.

[0041] II. Image-text cross-modal recognition model InstructDiffusion

[0042] Stable Diffusion is an efficient text-to-image generation model based on a denoising diffusion probabilistic model (DDPM). DDPM includes a forward diffusion process and a reverse diffusion process. In the forward diffusion process, random noise is gradually added to the sample data, and then the sample is generated from the noise through the reverse diffusion process. In the forward diffusion process, it is assumed that is the original data distribution, and given the unscrambled training sample , a sequence of scrambled data is obtained through the following Markov process :

[0043] (1)

[0044] where is based on the sample According to the Gaussian distribution, the next sample is generated by adding noise to the sample , t denotes the diffusion step, T is the number of diffusion steps; denotes the variance of the added noise when the corresponding diffusion step is t , which is used to control the noise intensity added at each step; denotes the identity matrix with the same dimension as ; denotes x obeys a normal distribution with mean and covariance .

[0045] Any sample can be calculated by equation (2), that is, by giving the initial sample and the variance , the sampling of the scrambled data at any diffusion step can be completed in one step:

[0046] (2)

[0047] where .

[0048] Using parameter renormalization, the sample can be sampled from :

[0049] (3)

[0050] From equation (3), we can see that if the variance , The distribution of can approximate the standard normal distribution quite well. .if If the value is much less than 1, then the reverse process applies. With forward process They have the same function form.

[0051] Utilizing the properties of the diffusion process, if from random Gaussian noise Start sampling, and the reverse process follows... You can get A new sample. However, because it cannot be obtained from... get Therefore, it is necessary to train a parameterized deep neural network. To approximate The input to this network is a scrambled image x. t and diffusion step t Learning and predicting the mean Covariance The denoised image x is obtained. t-1 .

[0052] The neural network is trained using maximum likelihood estimation, which enables it to predict the original image. Deep neural networks Assigned to each training sample The probability should be as high as possible, but It is difficult to calculate and needs to be done by minimizing the variational lower bound of the negative log-likelihood:

[0053] (4)

[0054] in, This represents the Kullback-Leibler divergence between two probability distributions.

[0055] Due to variance It has been fixed as a constant between 0 and 1, so L vlb The first item It is a constant; the second term This indicates that when the original image is introduced... When given known conditions, training the neural network makes each diffusion step... All of these can make As close as possible to the true posterior distribution of the forward diffusion process Due to parameterized deep neural networks To approximate The equation before parameter renormalization can be expressed as: And because of variance Set as a with Related constants and They are equal, expressed by the formula:

[0056] (5)

[0057] Therefore, trainable parameters only exist in the mean. In, and satisfy:

[0058] (6)

[0059] in, Indicates prediction Neural networks with moderate noise It is obtained by sampling through equation (3).

[0060] The objective function can be obtained from the above simplification. and reverse diffusion process The expressions (7) and (8), i.e., the noise prediction network The training objective is to minimize the prediction noise. and real noise The differences between them.

[0061] (7)

[0062] (8)

[0063] In Stable Diffusion, both diffusion and anti-diffusion occur in a low-dimensional latent space. Therefore, firstly, the encoder is generated by a pre-trained Variational Auto-Encoder (VAE). E (·) Mapping image data to latent variables z 0. This latent variable is usually a compressed representation of the original image space, used to carry the structural and texture information of the image. Then, the forward diffusion process gradually injects Gaussian noise into the latent variable, causing it to gradually degenerate from the real sample to a pure noise distribution. In the reverse diffusion process, the model learns a denoising network U-Net to gradually predict and remove noise, thereby realizing the recovery from random noise to structured latent variables.

[0064] The image-text cross-modal recognition model InstructDiffusion in this embodiment is built based on the diffusion model StableDiffusion. It transforms the visual task of anomaly recognition into image generation constrained by image conditions and instruction conditions, wherein the instruction conditions are provided by a language model. The task instruction is encoded to obtain a semantic embedding vector, which captures the semantic information of the text and provides semantic constraints for image generation. In the inverse diffusion stage, the latent variable of diffusion step T z T The feature is updated by the denoising U-Net, and the semantic embedding vector of the instruction is embedded into the multi-layer feature space of the U-Net through cross-attention Cross-Attention, and its calculation form is:

[0065]

[0066] Among them, the current layer image feature of the U-Net is taken as Q, which represents the feature query vector of the current image region to be "guided". The semantic embedding vector is taken as K and V, K represents the feature "index" of each semantic embedding vector, and V represents the feature "content value" of the corresponding semantic embedding vector. So that the model learns to focus on the semantic related area in the latent space, so as to realize language guided image generation and editing.

[0067] When the inverse diffusion process is completed T After step inverse reasoning, the final denoising latent variable is obtained , which is mapped back to the image space through the VAE decoder D (·) to obtain the target image, so as to realize the cross-modal generation from text to image and structural anomaly visualization.

[0068] The training target of the image-text cross-modal recognition model is as follows:

[0069] (10)

[0070] Among them, is the semantic embedding vector of the task instruction.

[0071] In the inference stage, starting from the standard Gaussian noise , based on the instruction condition embedding, through the learned denoising network , the inverse diffusion step is executed T times in turn, and finally the original image after denoising is obtained through the decoder D (·).

[0072] InstructDiffusion reformulates various computer vision tasks as conditional image-to-image generation problems constrained by natural language instructions. Based on Stable Diffusion, the training dataset consists of triplets (source image, target image, task instruction), where the task instruction precisely describes the semantic transformation relationship between the source image and the target image in natural language form; the visual model Stable Diffusion is responsible for performing semantic-level transformation on images according to these language instructions to generate corresponding target images; the mapping pair of "text semantics-image change" is formed, so that each training sample contains a semantically interpretable task relationship. The generated dataset is used to train the conditional diffusion network to predict the noise of the target latent variable. The Classifier-Free Guidance (CFG) technique is integrated to improve the semantic consistency between the generated output and the text editing instruction by introducing an image guidance coefficient s I and the text guidance coefficient s T Fine-grained control of the editing strength between image content and language instructions is achieved. Specifically represented as:

[0073] (11)

[0074] where, is the final noise predicted after comprehensive weighting, represents the visual condition obtained by encoding the input image through VAE, represents the semantic embedding vector of the task instruction. represents the noisy image latent representation in the diffusion process, represents the corresponding input, which is set to empty to obtain the unconditional branch, is the noise predicted under the unconditional condition, is the noise predicted only under the guidance of the image condition, is the noise predicted under the guidance of both the image and instruction conditions.

[0075] III. Model optimization

[0076] 1. CFG hyperparameter adjustment

[0077] The CFG hyperparameter adjustment step adopts an adaptive optimization algorithm, specifically a gradient descent method based on dynamic weight adjustment. This method takes the feature separation degree index as the optimization objective function, calculates the Euclidean distance of the abnormal region and the background region in the embedding space, and inputs this separation degree as a feedback signal into the optimizer.

[0078] The feature separation degree index is used to measure the distinguishability of the abnormal region and the background region in the model embedding space. Let the feature mean of the abnormal region be , the feature mean value of the background region is , the feature separation degree is defined as:

[0079] (12)

[0080] wherein, represents the Euclidean distance. The optimizer iteratively updates the weight of the text guidance coefficient s T and the image guidance coefficient s I The Adam optimization algorithm is used to constrain the weight update process to avoid oscillation and overfitting.

[0081] Further, when decreases, the optimizer increases the weight of s T ; when increases, the optimizer appropriately increases the weight of s I , thereby ensuring the balance between semantic analysis and image details.

[0082] Preferably, the text guidance coefficient s T is set to 5.0 to enhance the model's response to semantic instructions; the image guidance coefficient s I is set to 1.25 to maintain the detailed features in the image.

[0083] 2. Inference efficiency optimization

[0084] The inference efficiency optimization adopts an early stopping convergence strategy combined with a Pareto optimal solution set constraint to achieve a dynamic balance between accuracy and efficiency. During the inference process, first, a low sampling step is used to quickly generate an intermediate segmentation result, and multiple target performance indicators such as mean Pixel Accuracy (mPA), Dice coefficient, and IoU (Intersection over Union) are calculated. The system constructs a performance evaluation vector in a multi-objective optimization manner:

[0085] (13)

[0086] wherein, represents the inference time corresponding to the t-th sampling. The performance changes between consecutive sampling steps are monitored in real time, and the Pareto optimal condition is used as the early stopping criterion. When the following formula is satisfied, it is considered that the model has reached the Pareto convergence boundary:

[0087] (14)

[0088] wherein, is a performance change threshold (preferably set to 0.001-0.005) for determining whether the performance improvement tends to be saturated. When the above condition is met, the system triggers an early stopping mechanism to terminate the sampling process in advance, thereby significantly shortening the inference time while ensuring the recognition accuracy, achieving the optimal compromise between accuracy and efficiency.

[0089] Further, to avoid the degradation of segmentation performance caused by insufficient sampling steps, the system introduces a dynamic step control mechanism near the Pareto frontier region. In the early stage of convergence, a larger learning rate is used to quickly approach the optimal solution set, and the learning rate is gradually reduced when approaching the Pareto boundary, so that the performance curve converges smoothly to the stable interval. Preferably, the sampling step number is 50 steps to ensure that the minimum inference time is achieved while maintaining high structural consistency and segmentation accuracy.

[0090] Four, task instruction construction

[0091] The prompt framework includes a task instruction construction step, which adopts a double-layer structure of main instructions and prior knowledge embedding. The main instruction is used to clearly define the recognition or segmentation task performed by the model, and the prior knowledge is used to supplement morphological, textural and confusing feature information related to structural appearance anomalies, thereby improving the structural understanding ability and anomaly discrimination robustness of the model.

[0092] The main instruction gives the task target in natural language form, and its template is "Mark the pixels of %object% in red, leaving the rest unchanged." Among them, %object% is the abnormal class word to be identified, such as "crack regions", "corrosion spots", "spalling regions" or "exposed rebars".

[0093] Both the main instruction and the prior knowledge are converted into semantic vectors through the text encoder of the language model, wherein the main instruction encoding generates a task semantic vector C T , and the prior knowledge encoding generates a weak prior semantic vector E prior .

[0094] Unlike the unconditional branch in traditional CFG, which simultaneously empties the text condition and the image condition , the embodiment adopts a weak prior injection strategy, only introducing a weak prior semantic vector at the position of the empty text condition, and rewriting the branch as: to make the model retain low-intensity perception of abnormal morphology in the reverse diffusion process. This vector is only introduced in the unconditional branch to provide semantic context constraints, while other branches with image or text conditions remain unchanged to avoid replacing or interfering with the semantic information of the main task instructions. Therefore, the noise of the final prediction is expressed as

[0095] (15)

[0096] where, is the final noise of the integrated weighted prediction, represents the visual condition obtained by encoding the input image through VAE, represents the task semantic vector generated after encoding the main instruction. represents the noisy image latent representation in the diffusion process, represents the corresponding input that is nullified to obtain the unconditional branch, is the noise predicted only under the guidance of the weak prior semantic vector condition, is the noise predicted only under the guidance of the image condition, is the noise predicted under the guidance of both image and text conditions.

[0097] This weak prior injection strategy reduces the semantic difference distance between the unconditional branch and the conditional branch without changing the original CFG gradient algebraic structure, enabling the reverse diffusion denoising sampling to obtain more stable abnormal region focusing ability.

[0098] Taking crack identification as an example, first construct the main instruction prompt, such as “Mark the pixels of crack regions in red, leaving the rest unchanged.”, and extract it into a task semantic vector to clearly define the model output target and guide the U-Net to perform explicit labeling of crack regions during the reverse diffusion process. In addition, use prior knowledge to construct a weak semantic text prompt, such as “Cracks are thin and irregular in shape with distinct contrast from background, please avoid confusing with other slender structures.”, which is extracted into a weak prior semantic vector by a text encoder, and injected into the text input position of the unconditional branch , i.e., replacing with to give the model low-intensity semantic perception ability of crack morphology and texture.

[0099] The main instruction and the priori knowledge are involved in information fusion in the cross attention module of the U-Net, wherein the task semantics determine the editing direction, and the priori semantics provide the abnormal morphology background constraint, so that the visual features and the semantic knowledge are cooperatively guided.

[0100] V. Abnormal detection result and quantitative evaluation

[0101] In the abnormal detection process, the preprocessed to-be-recognized appearance image and the constructed task instruction are input into the InstructDiffusion model, so that the segmented structure abnormal image is output.

[0102] The output result is binarized to intuitively see the abnormal area and morphology. Further, by calculating the quantitative parameters such as length, width and area of the abnormal area, the extension function from qualitative identification to quantitative evaluation is realized, and data support is provided for structure health diagnosis and subsequent maintenance.

[0103] Embodiment 2

[0104] In an embodiment of the present application, a structure appearance abnormal cross-modal recognition system based on a diffusion model is provided, comprising:

[0105] An image acquisition module configured to acquire a to-be-recognized appearance image of a target structure;

[0106] An instruction construction module configured to construct an abnormal recognition instruction for a to-be-recognized abnormal type;

[0107] An abnormal recognition module configured to input the to-be-recognized appearance image and the abnormal recognition instruction into a trained image-text cross-modal recognition model to execute an abnormal recognition visual task and obtain a target image containing a semantic segmentation result;

[0108] The image-text cross-modal recognition model is constructed based on a diffusion model, and converts the abnormal recognition visual task into image generation constrained by an image condition and an instruction condition.

[0109] Embodiment 3

[0110] In an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the structure appearance abnormal cross-modal recognition method based on a diffusion model.

[0111] Embodiment 4

[0112] In an embodiment of the present application, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the method for identifying structural appearance anomaly cross-modal based on diffusion model.

[0113] Embodiment 5

[0114] In an embodiment of the present application, an electronic device is provided, comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the method for identifying structural appearance anomaly cross-modal based on diffusion model.

[0115] The present application is described in reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks. Figure 1 The function specified in one block or multiple blocks.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable data processing device to produce a computer-implemented process, so that the instructions executed by the computer or other programmable data processing device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks. Figure 1 The function specified in one block or multiple blocks.

[0117] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A cross-modal identification method for structural appearance anomalies based on a diffusion model, characterized in that, include: Obtain the appearance image of the target structure to be identified; For each type of anomaly to be identified, construct anomaly identification instructions; The appearance image to be identified and the anomaly identification instructions are input into the trained image-text cross-modal recognition model to perform the visual task of anomaly identification and obtain the target image containing semantic segmentation results. The image-text cross-modal recognition model is constructed based on a diffusion model, which transforms the visual task of anomaly recognition into image generation constrained by image conditions and instruction conditions. The anomaly identification instruction consists of a main instruction and prior knowledge. The main instruction embeds the anomaly type to be identified into the main instruction template. The prior knowledge includes the geometric shape, texture features, color contrast, and common confounding factors of the anomaly type to be identified. Both the main instructions and prior knowledge are transformed into semantic vectors by the text encoder of the language model, with the main instructions being encoded to generate the task semantic vector. C T Prior knowledge is encoded to generate weak prior semantic vectors. E prior ; A weak prior injection strategy is adopted, introducing weak prior semantic vectors only at empty condition locations in the text. This allows the cross-modal image-text recognition model to retain a low-intensity perception of abnormal morphologies during the back-diffusion process. Specifically: The weak prior semantic vector is introduced only in the completely unconditional branch to provide semantic background constraints. Branches with image or text conditions retain their original structure. The noise of the final prediction is expressed by the formula: in, The final noise predicted after comprehensive weighting, This represents the visual conditions obtained by encoding the input image using VAE. This indicates that the task semantic vector is generated after the main instruction is encoded. This represents the latent representation of a noisy image during the diffusion process. This indicates that the corresponding input is set to empty, thus obtaining an unconditional branch. This refers to noise predicted under conditions guided only by weak prior semantic vectors. For noise predicted solely by image conditions, For noise predicted under dual conditions of image and text, t Indicates the diffusion step. S I Represents the image guiding coefficient. S T This represents the text guidance coefficient.

2. The cross-modal identification method for structural appearance anomalies based on a diffusion model as described in claim 1, characterized in that, The image-text cross-modal recognition model adopts the Stable Diffusion model, an efficient text-guided image generation model based on the diffusion probability model. It includes a forward diffusion process and a backward diffusion process. First, random noise is gradually added to the sample data during the forward diffusion process, and then samples are generated from the noise through the backward diffusion process.

3. The cross-modal identification method for structural appearance anomalies based on a diffusion model as described in claim 2, characterized in that, The process of generating samples from noise through reverse diffusion involves using a trained denoising network to predict the noise at each diffusion step and then using the predicted noise for denoising.

4. The cross-modal identification method for structural appearance anomalies based on a diffusion model as described in claim 3, characterized in that, The denoising network employs a classifier-free guidance technique, introducing image guidance coefficients and instruction guidance coefficients into existing denoising networks. The instruction guidance coefficients are used to enhance the denoising network's response to semantic instructions, while the image guidance coefficients are used to preserve the detailed features in the image.

5. The cross-modal identification method for structural appearance anomalies based on a diffusion model as described in claim 4, characterized in that, It also includes using an adaptive optimization algorithm, with the goal of maximizing feature separation, to iteratively update the image guidance coefficient and the instruction guidance coefficient, ensuring a balance between semantic parsing and image details.

6. A cross-modal identification system for structural appearance anomalies based on a diffusion model, employing the method described in claim 1, characterized in that, include: The image acquisition module is configured to acquire the appearance image of the target structure to be identified. The instruction building module is configured to: build exception recognition instructions for the exception type to be identified; The anomaly detection module is configured to: input the appearance image to be identified and the anomaly detection instruction into the trained image-text cross-modal recognition model, perform the visual task of anomaly detection, and obtain the target image containing the semantic segmentation result; The image-text cross-modal recognition model is constructed based on a diffusion model, which transforms the visual task of anomaly recognition into image generation constrained by image conditions and instruction conditions.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-modal identification method for structural appearance anomalies based on a diffusion model as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a cross-modal identification method for structural appearance anomalies based on a diffusion model as described in any one of claims 1-5.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a cross-modal identification method for structural appearance anomalies based on a diffusion model as described in any one of claims 1-5.

Images