An image anomaly segmentation method based on semantic enhancement variational autoencoder

Abstract

The application relates to an image analysis method, in particular to an image anomaly segmentation method based on a semantic enhancement variational autoencoder. The method comprises a training stage and a testing stage. In the training stage, a semantic enhancement variational autoencoder model is constructed by introducing image and feature space distribution constraints to depict normal image distribution. In the testing stage, an abnormal attention map of a fusion channel domain and a spatial domain is calculated to capture global and local abnormal changes and segment abnormal regions in an abnormal image which are obviously different from normal images.

Description

Technical Field

[0001] This invention relates to image analysis methods, and in particular to an image anomaly segmentation method based on a semantically enhanced variational autoencoder. Background Technology

[0002] Currently, in fields such as medical imaging lesion detection and industrial image defect detection, we often face the challenge of extremely irregular and incomplete collection of abnormal samples. Furthermore, accurately labeling the abnormal regions of these samples requires not only specialized knowledge but is also very time-consuming. In contrast, normal samples are easier to collect and exhibit a more regular distribution. Therefore, unsupervised anomaly segmentation methods have become a key research focus in related fields.

[0003] Specifically, unsupervised image anomaly segmentation methods refer to methods that train a model using only normal samples and then segment anomalous regions in anomalous samples that do not match the learned normal samples during the inference phase. This type of method not only eliminates the dependence on difficult-to-collect anomalous samples but can also be used to segment rare anomalous regions. Currently, with the development of deep learning, deep learning-based unsupervised image anomaly segmentation methods are becoming a research hotspot in the field.

[0004] Existing unsupervised image anomaly segmentation methods fall into two main categories: reconstruction-based unsupervised image anomaly segmentation methods and representation-based unsupervised image anomaly segmentation methods, among which:

[0005] The reconstruction-based unsupervised image anomaly segmentation method is based on an encoder-decoder structure. It learns the distribution of normal images in a self-supervised manner and uses the reconstruction error of the input image and the reconstructed image to calculate pixel-level anomaly scores for anomaly region segmentation.

[0006] Representation-based unsupervised image anomaly segmentation methods use pre-trained neural networks to extract feature vectors or feature blocks representing images, and calculate pixel-level anomaly scores by measuring the distance between the feature vectors or feature blocks of the test image and the feature vectors or feature blocks of normal samples, thereby segmenting the abnormal regions.

[0007] However, both reconstruction-based and representation-based unsupervised image anomaly segmentation methods have many problems: for images with drastic changes in content structure, it is difficult to characterize the distribution of normal images. At the same time, the above methods are sensitive to image noise and it is difficult to take into account both global and local anomaly changes. Summary of the Invention

[0008] The purpose of this invention is to provide an image anomaly segmentation method based on a semantically enhanced variational autoencoder, thereby addressing the problems mentioned in the background section. To achieve the above objective, this invention provides the following technical solution:

[0009] An image anomaly segmentation method based on semantically enhanced variational autoencoder includes a training phase and a testing phase. The training phase involves constructing a semantically enhanced variational autoencoder model by introducing image and feature space distribution constraints to characterize the distribution of normal images. The testing phase involves calculating and fusing channel domain and spatial domain anomaly attention maps to segment anomaly regions in the anomaly image that are significantly different from the learned normal image.

[0010] The training phase steps include:

[0011] a) Collect image data and preprocess the collected data;

[0012] b) Using a traditional variational autoencoder, learn the distribution of a normal image;

[0013] c) Construct spatial distribution constraints for images to generate normal images rich in semantic information;

[0014] d) Construct feature space distribution constraints and train an encoder that accurately extracts features from normal images;

[0015] The testing phase steps include:

[0016] e) Introduce channel-domain anomaly attention to detect global anomalies;

[0017] f) Propose a spatial domain anomaly attention mechanism to detect local anomalies;

[0018] g) Merge the channel domain and spatial domain anomaly attention maps to construct pixel-level anomaly scores and segment the anomaly regions.

[0019] Preferably, in step a), the method for collecting image data includes:

[0020] a-1) Acquire image x using an imaging device;

[0021] a-2) The spatial resolution of the image is standardized by using bilinear interpolation.

[0022] Preferably, in step b), the conventional variational autoencoder includes an encoding module M. E Reparameter module M R and Reconstruction Module M D The encoding module of a traditional variational autoencoder starts from the input image x∈R M×N×C Extracting image features, the reparameter module is used to fit image features z∈R D The distribution of image features is used to reconstruct the image into a new image.

[0023] Preferably, in step c), the step of constructing image spatial distribution constraints includes:

[0024] c-1) Construct the semantic enhancement module M A The features z of the input image are shifted along random directions to obtain semantically enhanced features.

[0025] c-2) will Input Reconstruction Module M D This allows it to be reconstructed into a new, normal image with different semantic information.

[0026] c-3) Construct the image discrimination module M I The image discrimination module M I The optimization objective is:

[0027]

[0028] c-4) Aligning the newly generated image Distribution enhancement and reconstruction module M of the original image x D The ability to generate normal images, reconstruction module M D The optimization objective is:

[0029]

[0030] in, Features for semantic enhancement.

[0031] Preferably, in step d), the step of constructing the feature space distribution constraint includes:

[0032] d-1) The newly generated normal image Input the encoding module M sequentially E And weight parameter module M R , obtain features

[0033] d-2) Constructing the feature discrimination module M F By aligning the features of the newly generated normal image The distribution of the original image features z enhances the encoder's ability to extract normal image features. Encoder M E The optimization objective is:

[0034]

[0035] The feature discrimination module M F The optimization objective is:

[0036]

[0037] The reparameter module M R The optimization objective is:

[0038]

[0039] The overall optimization objective of the constructed semantically enhanced variational autoencoder model is:

[0040]

[0041] in, λ1, λ2, and λ3 are the optimization objectives of the variational autoencoder; λ1, λ2, and λ3 are hyperparameters used to balance the influence of different parts, and are usually set to λ1 = 1, λ2 = 1, and λ3 = 0.1.

[0042] Preferably, in step e), the channel domain anomaly attention step includes:

[0043] e-1) The image to be detected x t ∈R M×N×C Input encoding module M E Obtain the output feature map F∈R of its last convolutional block. w×h×c ;

[0044] e-2) Gaussian noise Add it to the image to be detected, and input it sequentially into the encoding module M. E and heavy parameter module M R To obtain its latent variable z n ;

[0045] e-3) Calculate the feature map F∈R w×h×c Feature map per channel F j ∈R w×h The abnormal weights are:

[0046]

[0047] in, For the feature map of layer j Feature map after being set to zero Input into reparameter module M R The latent variables obtained.

[0048] e-4) Channel domain anomaly attention map is as follows:

[0049]

[0050] Among them, F j ∈R w×h This is the feature map of the j-th channel.

[0051] Preferably, in step f), the spatial domain anomaly attention step includes:

[0052] f-1) The image to be detected x t Input the encoding module M sequentially E Reparameter module M R and Reconstruction Module M D Obtain the reconstructed image

[0053] f-2) The image to be detected and the reconstructed image are respectively sent to the encoding module M. E Obtain the encoding module M E Output feature maps of convolutional blocks at different scales and Where l = {1, 2, 3, 4} are convolutional blocks of different scales;

[0054] f-3) The multi-scale feature maps of the image to be detected and the reconstructed image are normalized in terms of length and width using adaptive average pooling, and then concatenated in the channel dimension to obtain multi-scale fused features. and Where w m and h m c represents the length and width dimensions of the largest scale feature. m This is the sum of the dimensions of the feature channels at different scales;

[0055] f-4) Spatial domain anomaly attention map is shown;

[0056]

[0057] Where (r,s)∈([0,R),[0,S)) represents the row and column positions of the multi-scale fused features, M (r,s) and The feature vector representing the position (r, s).

[0058] Preferably, in step g), the step of fusing channel domain and spatial domain anomaly attention to construct pixel-level anomaly scores includes:

[0059] g-1) Use bilinear interpolation to standardize the channel domain anomaly attention map and the spatial domain anomaly attention map;

[0060] g-2) Weight the channel domain anomaly attention map and the spatial domain anomaly attention map to obtain the final pixel-level anomaly score:

[0061]

[0062] Wherein, λ is a hyperparameter used to balance the channel domain anomaly attention map and the spatial domain anomaly attention map, and is set to λ = 0.002;

[0063] g-3) Based on different datasets, set a threshold T to binarize the pixel-level anomaly scores to 0 or 1 to obtain the segmentation results of the anomaly region. The threshold T can be adjusted according to the dataset. Set T = 0.5.

[0064] Compared with the prior art, the present invention has the following outstanding advantages:

[0065] This invention introduces distribution constraints in both image space and feature space. During training, it continuously generates normal images with new semantic information such as color, background, and texture for model training, thereby enhancing the model's learning of normal image distributions. In the detection phase, it comprehensively captures global and local anomalies by fusing channel-domain and spatial-domain anomaly attention, efficiently and accurately segmenting anomaly regions that differ from the learned normal distribution. Even for images with drastic changes in content structure, it can still characterize the distribution of normal images. Furthermore, it can handle images with significant noise, taking into account both global and local anomaly variations. Detailed Implementation

[0066] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] An image anomaly segmentation method based on semantically enhanced variational autoencoder includes a training phase and a testing phase. The training phase involves constructing a semantically enhanced variational autoencoder model by introducing constraints on the distribution of image and feature space to characterize the distribution of normal images. The testing phase involves segmenting abnormal regions in abnormal images that are significantly different from the learned normal images by calculating and fusing channel domain and spatial domain anomaly attention maps.

[0068] The training phase steps include:

[0069] a) Collect image data and preprocess the collected data;

[0070] In step a), the method for collecting image data includes:

[0071] a-1) Acquire image x using an imaging device;

[0072] a-2) The spatial resolution of the image is standardized by using bilinear interpolation.

[0073] b) Using a traditional variational autoencoder, learn the distribution of a normal image;

[0074] In step b), the conventional variational autoencoder includes an encoding module M.E Reparameter module M R and Reconstruction Module M D The encoding module of a traditional variational automatic editor starts from the input image x∈R M×N×C Extracting image features, the reparameter module is used to fit image features z∈R D The distribution of image features is used to reconstruct the image into a new image.

[0075] c) Construct spatial distribution constraints for images to generate normal images rich in semantic information;

[0076] In step c), the step of constructing image spatial distribution constraints includes:

[0077] c-1) Construct the semantic enhancement module M A The features z of the input image are shifted along random directions to obtain semantically enhanced features.

[0078] c-2) will Input Reconstruction Module M D This allows it to be reconstructed into a new, normal image with different semantic information.

[0079] c-3) Construct the image discrimination module M I The image discrimination module M I The optimization objective is:

[0080]

[0081] c-4) Aligning the newly generated image Distribution enhancement and reconstruction module M of the original image x D The ability to generate normal images, reconstruction module M D The optimization objective is:

[0082]

[0083] in, Features for semantic enhancement.

[0084] d) Construct feature space distribution constraints and train an encoder that accurately extracts features from normal images;

[0085] In step d), the step of constructing the feature space distribution constraints includes:

[0086] d-1) The newly generated normal image Input the encoding module M sequentially E And weight parameter module M R , obtain features

[0087] d-2) Constructing the feature discrimination module M F By aligning the features of the newly generated normal image The distribution of the original image features z enhances the encoder's ability to extract normal image features. Encoder M E The optimization objective is:

[0088]

[0089] The feature discrimination module M F The optimization objective is:

[0090]

[0091] The reparameter module M R The optimization objective is:

[0092]

[0093] The overall optimization objective of the constructed semantically enhanced variational autoencoder model is:

[0094]

[0095] in, λ1, λ2, and λ3 are the optimization objectives of the variational autoencoder; λ1, λ2, and λ3 are hyperparameters used to balance the influence of different parts, and are usually set to λ1 = 1, λ2 = 1, and λ3 = 0.1.

[0096] The testing phase steps include:

[0097] e) Introduce channel-domain anomaly attention to detect global anomalies;

[0098] In step e), the channel domain anomaly attention step includes:

[0099] e-1) The image to be detected x t ∈R M×N×C Input encoding module M E Obtain the output feature map F∈R of its last convolutional block. w×h×c ;

[0100] e-2) Gaussian noise Add it to the image to be detected, and input it sequentially into the encoding module M. E and heavy parameter module M R To obtain its latent variable z n ;

[0101] e-3) Calculate the feature map F∈R w×h×c Feature map per channel F j ∈R w×hThe abnormal weights are:

[0102]

[0103] in, For the feature map of layer j Feature map after being set to zero Input into reparameter module M R The latent variables obtained.

[0104] e-4) Channel domain anomaly attention map is as follows:

[0105]

[0106] Among them, F j ∈R w×h This is the feature map of the j-th channel.

[0107] f) Propose a spatial domain anomaly attention mechanism to detect local anomalies;

[0108] In step f), the spatial domain anomaly attention step includes:

[0109] f-1) The image to be detected x t Input the encoding module M sequentially E Reparameter module M R and Reconstruction Module M D Obtain the reconstructed image

[0110] f-2) The image to be detected and the reconstructed image are respectively sent to the encoding module M. E Obtain the encoding module M E Output feature maps of convolutional blocks at different scales and Where l = {1, 2, 3, 4} are convolutional blocks of different scales;

[0111] f-3) The multi-scale feature maps of the image to be detected and the reconstructed image are normalized in terms of length and width using adaptive average pooling, and then concatenated in the channel dimension to obtain multi-scale fused features. and Where w m and h m c represents the length and width dimensions of the largest scale feature. m This is the sum of the dimensions of the feature channels at different scales;

[0112] f-4) Spatial domain anomaly attention map is shown;

[0113]

[0114] Where (r, s) ∈ ([0, R), [0, S)) represents the row and column positions of the multi-scale fused features, M (r，s) and The feature vector representing the position (r, s).

[0115] g) Integrate channel domain and spatial domain anomaly attention to construct pixel-level anomaly scores and segment anomaly regions.

[0116] In step g), the step of fusing channel-domain and spatial-domain anomaly attention to construct pixel-level anomaly scores includes:

[0117] g-1) Use bilinear interpolation to standardize the channel domain anomaly attention map and the spatial domain anomaly attention map;

[0118] g-2) Weight the channel domain anomaly attention map and the spatial domain anomaly attention map to obtain the final pixel-level anomaly score:

[0119]

[0120] Wherein, λ is a hyperparameter used to balance the channel domain anomaly attention map and the spatial domain anomaly attention map, and is set to λ = 0.002;

[0121] g-3) Based on different datasets, set a threshold T to binarize the pixel-level anomaly scores to 0 or 1 to obtain the segmentation results of the anomaly region. The threshold T can be adjusted according to the dataset. Set T = 0.5.

[0122] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image anomaly segmentation method based on a semantically enhanced variational autoencoder, characterized in that: The model includes a training phase and a testing phase. The training phase involves constructing a semantically enhanced variational autoencoder model by introducing constraints on the distribution of image and feature space to characterize the distribution of normal images. The testing phase involves calculating and fusing channel domain and spatial domain anomaly attention maps to segment abnormal regions in abnormal images that are significantly different from the learned normal images. The training phase steps include: a) Collect image data and preprocess the collected data; b) Using a traditional variational autoencoder, learn the distribution of a normal image; c) Construct spatial distribution constraints for images to generate normal images rich in semantic information; d) Construct feature space distribution constraints and train an encoder that accurately extracts features from normal images; The testing phase steps include: e) Introduce channel-domain anomaly attention to detect global anomalies; f) Propose a spatial domain anomaly attention mechanism to detect local anomalies; g) Merge the channel domain and spatial domain anomaly attention maps to construct pixel-level anomaly scores and segment anomaly regions; In step b), the conventional variational autoencoder includes an encoding module M. E Reparameter module M R and Reconstruction Module M D The encoding module of the variational autoencoder starts from the input image x∈R M×N×C Extracting image features, the reparameter module is used to fit image features z∈R D The distribution of image features is used to reconstruct the image into a new image. In step c), the step of constructing image spatial distribution constraints includes: c-1) Construct the semantic enhancement module M A The features z of the input image are shifted along random directions to obtain semantically enhanced features. c-2) will Input Reconstruction Module M D This allows it to be reconstructed into a new, normal image with different semantic information. c-3) Construct the image discrimination module M I The image discrimination module M I The optimization objective is: c-4) Aligning the newly generated image Distribution enhancement and reconstruction module M of the original image x D The ability to generate normal images, reconstruction module M D The optimization objective is: in, Features for semantic enhancement; In step d), the step of constructing the feature space distribution constraints includes: d-1) The newly generated normal image Input the encoding module M sequentially E And weight parameter module M R , obtain features d-2) Constructing the feature discrimination module M F By aligning the features of the newly generated normal image The distribution of the original image features z enhances the encoder's ability to extract normal image features. Encoder M E The optimization objective is: The feature discrimination module M F The optimization objective is: The reparameter module M R The optimization objective is: The overall optimization objective of the constructed semantically enhanced variational autoencoder model is: in, λ1, λ2, and λ3 are the optimization objectives of the variational autoencoder; λ1, λ2, and λ3 are hyperparameters used to balance the influence of different parts, and are usually set to λ1 = 1, λ2 = 1, and λ3 = 0.

1. In step e), the channel domain anomaly attention step includes: e-1) The image to be detected x t ∈R M×N×C Input encoding module M E Obtain the output feature map F∈R of its last convolutional block. w×h×c ; e-2) Gaussian noise Add it to the image to be detected, and input it sequentially into the encoding module M. E and heavy parameter module M R To obtain its latent variable z n ; e-3) Calculate the feature map F∈R w×h×c Feature map per channel F j ∈R w×h The abnormal weights are: in, For the feature map of layer j Feature map after being set to zero Input into reparameter module M R The latent variables obtained; e-4) Channel domain anomaly attention map is as follows: Among them, F j ∈R w×h This is the feature map of the j-th channel; In step f), the spatial domain anomaly attention step includes: f-1) The image to be detected x t Input the encoding module M sequentially E Reparameter module M R and Reconstruction Module M D Obtain the reconstructed image f-2) The image to be detected and the reconstructed image are respectively sent to the encoding module M. E Obtain the encoding module M E Output feature maps of convolutional blocks at different scales and Where l = {1,2,3,4} represents convolutional blocks of different scales; f-3) The multi-scale feature maps of the image to be detected and the reconstructed image are normalized in terms of length and width using adaptive average pooling, and then concatenated in the channel dimension to obtain multi-scale fused features. and Where w m and h m c represents the length and width dimensions of the largest scale feature. m This is the sum of the dimensions of the feature channels at different scales; f-4) Spatial domain anomaly attention map is shown; Where (r,s)∈([0,R),[0,S)) represents the row and column positions of the multi-scale fused features, M (r,s) and The feature vector representing the position (r, s).

2. The image anomaly segmentation method based on semantically enhanced variational autoencoder according to claim 1, characterized in that: In step a), the method for collecting image data includes: a-1) Acquire image x using an imaging device; a-2) The spatial resolution of the image is standardized by using bilinear interpolation.

3. The image anomaly segmentation method based on semantically enhanced variational autoencoder according to claim 1, characterized in that: In step g), the step of fusing the channel domain and spatial domain anomaly attention maps to construct pixel-level anomaly scores includes: g-1) Use bilinear interpolation to standardize the channel domain anomaly attention map and the spatial domain anomaly attention map; g-2) Weight the channel domain anomaly attention map and the spatial domain anomaly attention map to obtain the final pixel-level anomaly score: Wherein, λ is a hyperparameter used to balance the channel domain anomaly attention map and the spatial domain anomaly attention map, and is set to λ = 0.002; g-3) Based on different datasets, set a threshold T to binarize the pixel-level anomaly scores to 0 or 1 to obtain the segmentation results of the anomaly regions. The threshold T is adjusted according to the dataset, and T = 0.5 is set.

Images