A method for constructing an image anomaly detection model based on a mask multi-modal generative adversarial network

Abstract

The application discloses a kind of construction methods of image anomaly detection model based on mask multimodal generative adversarial network, it is related to image anomaly detection field.The existing encoder model has large capacity, the feature difference between defect and normal area is not obvious;The method of image reconstruction idea mostly only considers the single mode of image, cannot fully utilize the correlation between different modalities of image data.The application provides the following scheme, acquires data: the public image dataset is divided into training dataset and validation dataset;Data processing: the adversarial network of training dataset is divided into a, b two main branches, and the above sum is obtained The image to be reconstructed required in generator of generative adversarial network;Network model framework is constructed;Design model loss function;Training and testing network, also applicable to image detection technology field.

Description

Technical Field

[0001] This invention relates to the field of image anomaly detection. Background Technology

[0002] Anomaly detection is a well-known computer vision task used to detect anomalous features in a given image. A large portion of traditional anomaly detection algorithms fall under the category of supervised learning, requiring the collection and accurate annotation of sufficient target category samples. However, in many application scenarios, collecting a sufficient number of samples is difficult, and the scarcity of defect samples, annotation costs, and lack of prior knowledge about defects can cause supervised methods to fail. Defects are irregular in shape and vary greatly in size, making it difficult for detection boxes to accurately represent their location, and they easily introduce a lot of irrelevant background information, complicating the evaluation of defect detection performance.

[0003] Unsupervised image anomaly detection methods typically only require normal, defect-free samples for network training. These methods focus more on the features of normal samples; when anomalies are detected during defect detection, a defect is considered detected. This approach does not require a large amount of defective scene image data and can effectively solve the problem of imbalanced samples.

[0004] In unsupervised image anomaly detection, image reconstruction-based methods can achieve good results. These methods train the model using only normal samples, allowing it to learn the feature distribution of normal samples. Because the model's parameters are only trained on normal samples, it can only reconstruct normal samples well and is sensitive to anomalies, where defective regions can cause significant reconstruction errors. Autoencoder (AE) methods employ an encoder-decoder network structure. The encoder encodes the input image into a latent space representation, and the decoder uses this latent space representation to reconstruct the image. Defect localization is achieved based on the reconstruction error between the input and reconstructed images. However, autoencoders (AEs) exhibit blurring during reconstruction, which can easily lead to false detections of normal pixels when calculating the reconstruction error. Generative Adversarial Networks (GANs) consist of a generator and a discriminator; the adversarial process between the generator and discriminator can improve reconstruction capabilities. When the training set consists entirely of normal samples, the model can only learn the distribution of normal samples; therefore, the latent space variables, after passing through the generator, only produce normal images. Similarly, anomaly detection is determined based on the reconstruction error between the test image and the reconstructed image. However, image reconstruction methods still have limitations. Even when trained only on normal samples, the model may still completely reconstruct unseen defects. This phenomenon is mainly due to two reasons: the large capacity of models such as auto-encoders (AEs) and the lack of significant feature differences between defects and normal regions. Currently, most image reconstruction methods only consider a single modality of the image and fail to fully utilize the correlations between different modalities of image data. Summary of the Invention

[0005] To address the issues that existing auto-encoder (AE) models have large capacity but lack significant feature differences between defective and normal regions; and that current image reconstruction methods mostly consider only a single modality of the image, failing to fully utilize the correlation between different modalities of image data, this invention provides the following technical solution:

[0006] This invention provides a method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks, comprising the following steps:

[0007] S1: Data Acquisition: Acquire a publicly available image dataset and divide the image dataset into a training dataset and a validation dataset; the training dataset is used to optimize the parameters in the adversarial network; the validation dataset is used to evaluate the generalization ability of the model.

[0008] S2: Data Processing: The adversarial network of the training dataset in S1 is divided into two main branches, a and b. The training image samples I located in branch a are processed through a frequency domain decoupling module to obtain components I1 and I2 in different frequency domains. The two frequency domain branches I1 and I2 are masked using the same mask rate to obtain... The training image sample I on branch b is masked using a different mask rate than that on branch a. The and To generate the image to be reconstructed required by the generator in the adversarial network;

[0009] S3: Construct the network model framework; Based on the data processing process described in S2, use a discriminator to improve the generator's reconstruction capability;

[0010] S4: Design the model loss function; Based on the network model framework constructed in S3, design three loss functions: generator loss function, discriminator loss function, and discriminator latent layer loss function;

[0011] S5: Training and testing the network; train the model using the training set, optimize the network parameters using the loss function, backpropagation algorithm, and stochastic gradient descent algorithm obtained from S4, and stop iterating when the model performs best on the test dataset.

[0012] Furthermore, a preferred embodiment is provided, wherein S2 specifically comprises:

[0013] In branch S2.1, the images in the training dataset are scaled to 256×256 pixels, and the scaled images are passed through the frequency decoupling module to obtain two frequency components I1 and I2.

[0014] S2.2 scales I1 and I2 to 32×32 pixels and converts them to the range [0, 1], then performs a random masking with the same masking rate to obtain...

[0015] S2.3 In branch b, the images in the training dataset are scaled to 32×32 pixels and converted to the range [0, 1]. The scaled images are then masked using a random mask rate different from that in step 2.2.

[0016] Furthermore, in a preferred embodiment, step S2.1 of obtaining the two frequency components I1 and I2 specifically involves:

[0017] The original image I is Gaussian smoothed, and then downsampled by discarding even-numbered rows and columns, reducing the entire image I to one-quarter of its original size, thus obtaining the final image.

[0018]

[0019] In the formula:

[0020]

[0021] Then the image The rows and columns are doubled in sampling, and then Gaussian smoothing is performed to obtain image I1:

[0022]

[0023] Based on the original image I and the image The difference is I2, that is:

[0024]

[0025] Furthermore, a preferred embodiment is provided, wherein the method for obtaining the generator loss function, discriminator loss function, and discriminator latent layer loss function in S4 is as follows:

[0026] S4.1 Generator Loss Function: The image to be reconstructed is input into the generator to obtain the reconstructed image and the reconstruction error. The model parameters are adjusted by optimizing the reconstruction error so that the reconstructed image is close to the original image. The reconstruction error is the generator's loss function.

[0027] S4.2 The discriminator's loss function is used for adversarial training; the discriminator reconstructs the image from branch a. Reconstructed image of branch b Reconstructing distinctions;

[0028] S4.3 Discriminator: Shallow layer loss function, latent layer loss penalizes the reconstructed image in the latent space. and Similarity between them; use the features of the last convolutional layer of discriminator D as latent spatial features.

[0029] Furthermore, a preferred embodiment is provided, wherein S4 specifically comprises:

[0030] S4.1 Generator Loss Function: The image to be reconstructed is input into the generator to obtain the reconstructed image and the reconstruction error. The model parameters are adjusted by optimizing the reconstruction error so that the reconstructed image is close to the original image. The reconstruction error is the generator's loss function.

[0031] S4.2 The discriminator's loss function is used for adversarial training;

[0032] S4.3 Discriminator: Shallow layer loss function, latent layer loss penalizes the reconstructed image in the latent space. and Similarity between them; using features from the last convolutional layer of discriminator D as latent spatial features;

[0033] S4.4 Design the total loss function of the network, which combines the above three loss functions to form the total loss function of the model.

[0034] Furthermore, a preferred embodiment is provided, wherein the loss function in S4.1 is:

[0035] The loss function is defined as the reconstructed image The mean absolute error between each element in the original image I is given by the following formula:

[0036]

[0037] in, To reconstruct the loss, Let be the expected value of the I-distribution.

[0038] Furthermore, in a preferred embodiment, the loss function of the discriminator in S4.2 is:

[0039]

[0040] in, To combat the losses, for The expected value of the distribution. for The expected value of the distribution.

[0041] Furthermore, a preferred embodiment is provided in which the latent loss in S4.3 is used to reconstruct the image in the latent space. and The similarity between them is in, This refers to the discriminator's potential loss.

[0042] Furthermore, a preferred embodiment is provided, wherein the total network loss function designed in S4.4 is:

[0043] in and Let be the reconstruction losses for branches a and b, respectively. It is to combat losses. To balance the training process, a weight parameter λ is introduced for the discriminator's latent loss. con =50, λ adv =1,λ lat =1.

[0044] Furthermore, a preferred embodiment is provided, wherein S5 specifically comprises:

[0045] S5.1 After processing the training dataset in S2, it is fed into the network model framework constructed in S3. The reconstruction loss, adversarial loss, and discriminator latent feature loss are calculated according to the model loss function designed in S4. The adversarial loss is obtained by backpropagation algorithm to obtain the gradient of each network layer of the discriminator, and the parameters of each network layer of the discriminator are optimized by stochastic gradient descent algorithm. The total loss function is obtained by backpropagation algorithm to obtain the gradient of each network layer of the generator, and the parameters of each network layer of the generator are optimized by stochastic gradient descent algorithm. This is one round of training model operation.

[0046] S5.2 The image to be reconstructed is fed into the trained model, and anomalies are detected by comparing the anomaly score with the actual detected scene. The anomaly score is:

[0047]

[0048] in and Let be the reconstruction losses for branches a and b, respectively. λ is the discriminator latent loss, and λ is the weight parameter set to 0.9.

[0049] The anomaly score for each test sample in the test set is obtained; the set of anomaly scores is A, and A is scaled to [0, 1]. The final anomaly score of the image to be reconstructed is:

[0050]

[0051] The advantages of this invention are:

[0052] To overcome the inherent characteristics of industrial image datasets and the shortcomings of anomaly detection techniques, this invention proposes a method for constructing an image anomaly detection model based on a masked multimodal generative adversarial network. This method uses multi-scale feature fusion to address the issue of varying defect sizes and shapes in anomaly samples. A design with different mask rates is derived from a masked autoencoder, adding masks to normal images during training to simulate defects. The network model is then trained to reconstruct the corresponding original image. The trained model possesses the ability to eliminate defects based on context, preventing the model from reconstructing unseen defects effectively. Furthermore, considering the potential distributional correlations between different image modalities, frequency domain information and RGB image information are used for adversarial training to improve detection performance.

[0053] The model constructed in this invention is applicable to the field of image detection technology. Attached Figure Description

[0054] Figure 1 This is a frequency decoupling diagram illustrating the construction method of the image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method 1.

[0055] Figure 2 This is a mask diagram illustrating the construction method of the image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method 3.

[0056] Figure 3 This is a schematic diagram of the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks as described in Implementation Method 4.

[0057] Figure 4 This is a flowchart illustrating the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks as described in Implementation Method 10. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0059] Implementation Method 1: This implementation method provides a method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks, including the following steps:

[0060] S1: Data Acquisition: Acquire a publicly available image dataset and divide the image dataset into a training dataset and a validation dataset; the training dataset is used to optimize the parameters in the adversarial network; the validation dataset is used to evaluate the generalization ability of the model.

[0061] S2: Data Processing: The adversarial network of the training dataset in S1 is divided into two main branches, a and b. The training image samples I located in branch a are processed through a frequency domain decoupling module to obtain components I1 and I2 in different frequency domains. The two frequency domain branches I1 and I2 are masked using the same mask rate to obtain... The training image sample I on branch b is masked using a different mask rate than that on branch a. The and To generate the image to be reconstructed required by the generator in the adversarial network;

[0062] S3: Construct the network model framework; Based on the data processing process described in S2, use a discriminator to improve the generator's reconstruction capability;

[0063] S4: Design the model loss function; Based on the network model framework constructed in S3, design three loss functions: generator loss function, discriminator loss function, and discriminator latent layer loss function;

[0064] S5: Training and testing the network; train the model using the training set, optimize the network parameters using the loss function, backpropagation algorithm, and stochastic gradient descent algorithm obtained from S4, and stop iterating when the model performs best on the test dataset.

[0065] See Figure 1 The implementation method is described below. The publicly available image dataset described in this implementation method is obtained from a public database. The training dataset in S1 is sample data used for model fitting and is used to optimize various parameters in the network. The verification dataset is used to evaluate the generalization ability of the final model and does not participate in model training and parameter optimization.

[0066] This implementation method utilizes a masked multimodal generative adversarial network-based image anomaly detection model to address the challenge of resolving the varying sizes and shapes of defects in anomalous samples through multi-scale feature fusion. Furthermore, it employs the MVTecAD anomaly detection dataset as an example. The MVTecAD dataset mimics a real-world industrial production scenario and contains 5354 high-resolution color images of different targets and textures. It includes normal images (those without defects) used for training, and anomalous images used to apply the constructed image anomaly detection model. The anomalous images exhibit 70 different types of defects, such as scratches, dents, contamination, and various structural variations.

[0067] Implementation Method Two: This implementation method further defines the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method One. Specifically, S2 is as follows:

[0068] In branch S2.1, the images in the training dataset are scaled to 256×256 pixels, and the scaled images are passed through the frequency decoupling module to obtain two frequency components I1 and I2.

[0069] S2.2 scales I1 and I2 to 32×32 pixels and converts them to the range [0, 1], then performs a random masking with the same masking rate to obtain...

[0070] S2.3 In branch b, the images in the training dataset are scaled to 32×32 pixels and converted to the range [0, 1]. The scaled images are then masked using a random mask rate different from that in step 2.2.

[0071] See Figure 2This embodiment further demonstrates the masking effect. Specifically, it uses a mask autoencoder to design input images with different mask rates in an image anomaly detection model. Masks are added to normal images during training to simulate defects, and the network model is trained to restore the corresponding original image. After training, the image anomaly detection model has the ability to eliminate defects based on context, preventing the model from reconstructing defects it has never seen before. Secondly, considering the potential distributional correlations between different image modalities, adversarial training using image frequency domain information and RGB image information is used to improve detection performance.

[0072] Implementation Method 3: This implementation method further defines the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method 1. Specifically, S2.1 obtaining the two frequency components I1 and I2 is as follows:

[0073] The original image I is Gaussian smoothed, and then downsampled by discarding even-numbered rows and columns, reducing the entire image I to one-quarter of its original size, thus obtaining the final image.

[0074]

[0075] In the formula:

[0076]

[0077] Then the image The rows and columns are doubled in sampling, and then Gaussian smoothing is performed to obtain image I1:

[0078]

[0079] Based on the original image I and the image The difference is I2, that is:

[0080]

[0081] See Figure 1 This implementation method is described below. Figure 1 This means that in branch a, the images in the dataset are scaled to 256×256 pixels. The scaled images are then processed by the frequency decoupling module to obtain two frequency components, I1 and I2, which are a comparison of the original image I before frequency decoupling and I1 and I2 after frequency decoupling.

[0082] Implementation Method 4: This implementation method further defines the construction method of the image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method 1. The method for obtaining the generator loss function, discriminator loss function, and discriminator latent layer loss function in S4 is as follows:

[0083] S4.1 Generator Loss Function: The image to be reconstructed is input into the generator to obtain the reconstructed image and the reconstruction error. The model parameters are adjusted by optimizing the reconstruction error so that the reconstructed image is close to the original image. The reconstruction error is the generator's loss function.

[0084] S4.2 The discriminator's loss function is used for adversarial training; the discriminator reconstructs the image from branch a. Reconstructed image of branch b Reconstructing distinctions;

[0085] S4.3 Discriminator: Shallow layer loss function, latent layer loss penalizes the reconstructed image in the latent space. and Similarity between them; use the features of the last convolutional layer of discriminator D as latent spatial features.

[0086] See Figure 3 This embodiment describes the design of the network's total loss function, which combines the three loss functions mentioned above to form the model's total loss function. In this embodiment, the U-Net architecture is based on an encoder-decoder model, where the encoder extracts features from the input image, and the decoder generates a segmentation map based on these features. However, U-Net also integrates skip connections between the encoder and decoder to preserve spatial information and improve segmentation accuracy. The encoder is essentially a feature extraction network that extracts high-level semantic information from the high-dimensional image and stores it in a low-dimensional encoding. The generator takes a 32×32 pixel image as input. After passing through convolutional and pooling layers, it produces five feature maps of different scales: 16×16, 8×8, 4×4, 2×2, and the original 32×32 input. The 2×2 feature map is then upsampled or deconvolved to obtain a 4×4 feature map. This 4×4 feature map is concatenated with the previous 4×4 feature map along its channels. The concatenated feature map is then convolved and upsampled to obtain an 8×8 feature map, which is then concatenated with the previous 8×8 feature map, convolved again, and upsampled once more. After five upsampling operations, a 32×32 prediction result with the same size as the input image is obtained. The convolutional module consists of ordinary convolutional layers, non-linear activation function layers, and batch normalization layers. The convolutional layers compute feature maps, the batch normalization layers normalize the data, reducing data jitter between batches and thus accelerating convergence and improving training speed, and the non-linear activation function layers add non-linear information to prevent the model from degenerating into a simple linear model. The process of obtaining a low-dimensional representation of data is called encoding, and this part is called the encoder. The process of restoring high-dimensional data from low-dimensional data is called decoding, and this part is called the decoder.

[0087] This implementation details the role of the discriminator, which is used to determine the similarity between the reconstructed images from branch a and branch b. Based on the discriminant results, the generator parameters are then optimized to make the reconstructed images from the two branches more similar, thus improving the generator's reconstruction capability. The two feature maps reconstructed from branch a are summed pixel-by-pixel as the first input to the discriminator, and the feature map from branch b is used as the second input. The discriminator's input is a 3×32×32 image, and its output is a 1×100×100 vector indicating the distribution of the input sample types. The discriminator consists of two basic modules, each containing a convolutional layer, a batch normalization layer, and a non-linear activation function layer.

[0088] Implementation Method 5: This implementation method further defines the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks as described in Implementation Method 4. The loss function in S4.1 is:

[0089] The loss function is defined as the reconstructed image The mean absolute error between each element in the original image I is given by the following formula:

[0090]

[0091] in, To reconstruct the loss, Let be the expected value of the I-distribution.

[0092] In this implementation, the image to be reconstructed from the image anomaly detection model is input into the generator to obtain the reconstructed image and the reconstruction error. The model parameters are adjusted by optimizing the reconstruction error to make the reconstructed image as close as possible to the original image. The reconstruction error is the generator's loss function, which is defined as the reconstructed image... The mean absolute error between each element in the original image I is described in detail in Implementation Methods VI, VII, VIII, and IX.

[0093] Implementation Method Six: This implementation method further defines the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method Five. The loss function of the discriminator in S4.2 is:

[0094]

[0095] in, To combat the losses, for The expected value of the distribution. for The expected value of the distribution.

[0096] Implementation Method Seven: This implementation method further defines the image anomaly detection model based on masked multimodal generative adversarial networks described in Implementation Method Five. Specifically, it describes the latent loss penalty in S4.3 for reconstructing the image in the latent space. and The similarity between them is in, This refers to the discriminator's potential loss.

[0097] Implementation Method Eight: This implementation method further defines the method for constructing an image anomaly detection model based on a masked multimodal generative adversarial network as described in Implementation Method Five. In S4.4, the total network loss function is designed as follows: in, and Let be the reconstruction losses for branches a and b, respectively. It is to combat losses. To balance the training process, a weight parameter λ is introduced for the discriminator's latent loss. con =50, λ adv =1,λ lat =1.

[0098] Implementation Method Nine: This implementation method further defines the method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks as described in Implementation Method One. Specifically, S5 is as follows:

[0099] S5.1 After processing the training dataset in S2, it is fed into the network model framework constructed in S3. The reconstruction loss, adversarial loss, and discriminator latent feature loss are calculated according to the model loss function designed in S4. The adversarial loss is obtained by backpropagation algorithm to obtain the gradient of each network layer of the discriminator, and the parameters of each network layer of the discriminator are optimized by stochastic gradient descent algorithm. The total loss function is obtained by backpropagation algorithm to obtain the gradient of each network layer of the generator, and the parameters of each network layer of the generator are optimized by stochastic gradient descent algorithm. This is one round of training model operation.

[0100] S5.2 The image to be reconstructed is fed into the trained model, and anomalies are detected by comparing the anomaly score with the actual detected scene. The anomaly score is:

[0101]

[0102] in and Let be the reconstruction losses for branches a and b, respectively. λ is the discriminator latent loss, and λ is the weight parameter set to 0.9.

[0103] The anomaly score for each test sample in the test set is obtained; the set of anomaly scores is A, and A is scaled to [0, 1]. The final anomaly score of the image to be reconstructed is:

[0104]

[0105] This implementation method feeds test image data into a trained image anomaly detection model and compares anomaly scores with a threshold. The threshold for anomaly scores can be set according to the actual detection scenario.

[0106] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks, characterized in that, Includes the following steps: S1: Data Acquisition: Acquire a publicly available image dataset and divide the image dataset into a training dataset and a validation dataset; The training dataset is used to optimize the parameters in the adversarial network; The validation set is used to evaluate the model's generalization ability; S2: Data Processing: Divide the adversarial network of the training dataset in S1 into two main branches, a and b. Process the training image samples located in branch a. Components in different frequency domains are obtained through a frequency domain decoupling module. , Using the same mask rate , Two frequency domain branch masks are obtained , Training image samples on branch b Using a different masking rate than that on branch A, we obtain... The To generate the image to be reconstructed required by the generator in the adversarial network; S3: Construct the network model framework; Based on the data processing process described in S2, use a discriminator to improve the generator's reconstruction capability; S4: Design the model loss function; Based on the network model framework constructed in S3, design three loss functions: generator loss function, discriminator loss function, and discriminator latent layer loss function; S5: Training and testing the network; train the model using the training set, optimize the network parameters based on the loss function, backpropagation algorithm, and stochastic gradient descent algorithm obtained from S4, and stop iterating when the model performs best on the test dataset. Specifically, S2 is: S2.1 In branch a, the images in the training dataset are scaled to 256×256 pixels, and the scaled images are passed through the frequency decoupling module to obtain two frequency components. and ; S2.2 will , Scale to 32×32 pixels and convert to the range [0, 1], then perform a random mask with the same mask rate to obtain... , ; S2.3 In branch b, scale the images in the training dataset to 32×32 pixels and convert them to the range [0, 1]. Then, apply a random mask to the scaled images using a different mask rate than that used in step 2.

2. .

2. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 1, characterized in that, S2.1 obtains two frequency components. and Specifically: For the original image Gaussian smoothing is applied, followed by downsampling by discarding even-numbered rows and columns, to refine the entire image. It becomes one-quarter of its original value, and the resulting image is obtained. : In the formula: Then the image The image is obtained by doubling the number of rows and columns sampled and then performing Gaussian smoothing. : Based on the original image With images The difference is ,Right now: 。 3. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 1, characterized in that, The methods for obtaining the generator loss function, discriminator loss function, and discriminator latent layer loss function described in S4 are as follows: S4.1 Generator Loss Function: The image to be reconstructed is input into the generator to obtain the reconstructed image and the reconstruction error. The model parameters are adjusted by optimizing the reconstruction error so that the reconstructed image is close to the original image. The reconstruction error is the generator's loss function. S4.2 The discriminator's loss function is used for adversarial training; the discriminator reconstructs the image from branch a. Reconstructed image of branch b Reconstructing distinctions; S4.3 Loss function for shallow layers of the discriminator, and loss function for latent layers penalizing the reconstructed image in the latent space. and Similarity between them; use the features of the last convolutional layer of discriminator D as latent spatial features.

4. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 3, characterized in that, The loss function in S4.1 is: The loss function is defined as the reconstructed image and the original image The mean absolute error between each element is given by the following formula: in, To reconstruct the loss, for The expected value of the distribution.

5. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 4, characterized in that, The loss function of the discriminator in S4.2 is: in To combat the losses, for The expected value of the distribution. for The expected value of the distribution.

6. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 5, characterized in that, The image reconstructed in the latent space in S4.3 is penalized by the latent loss. and The similarity between them is ,in, This refers to the discriminator's potential loss.

7. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 5, characterized in that, The total network loss function designed in S4.4 is as follows: in and The reconstruction losses for branches a and b are respectively, and weight parameters are introduced during balanced training. =50, =1, =1.

8. The method for constructing an image anomaly detection model based on masked multimodal generative adversarial networks according to claim 1, characterized in that, Specifically, S5 is: S5.1 After processing the training dataset in S2, it is fed into the network model framework constructed in S3. The reconstruction loss, adversarial loss, and discriminator latent feature loss are calculated according to the model loss function designed in S4. The adversarial loss is obtained by backpropagation algorithm to obtain the gradient of each network layer of the discriminator, and the parameters of each network layer of the discriminator are optimized by stochastic gradient descent algorithm. The total loss function is obtained by backpropagation algorithm to obtain the gradient of each network layer of the generator, and the parameters of each network layer of the generator are optimized by stochastic gradient descent algorithm. This is one round of training model operation. S5.2 The image to be reconstructed is fed into the trained model, and anomalies are detected by comparing the anomaly score with the actual detection scene. The anomaly score is: in and Let be the reconstruction losses for branches a and b, respectively. It is the discriminator's latent layer loss. The weight parameter is set to 0.9; The abnormal score is obtained for each test sample in the test set; the abnormal score set is... Scaling A to [0,1], the final anomaly score of the image to be reconstructed is: 。

Images