Few-sample image anomaly detection method and system based on image registration

Abstract

The present invention provides a method and system for detecting anomalies of a few samples of images based on image registration, including: extracting high-dimensional features of a supporting image and an image to be detected; performing spatial transformation of the high-dimensional features of the image to obtain transformed image features; implementing feature encoding for the transformed image features; implementing feature registration for the encoded features; fitting the feature distribution of the supporting image for the transformed image features to obtain a feature distribution model; and implementing image anomaly assessment for the transformed image features and the feature distribution model. In view of the problems existing in the current anomaly detection methods, the present invention proposes a method for detecting anomalies of a few samples based on image registration. The present invention uses data of objects of known categories to train a generalizable model, and does not need to retrain the model for data of objects of new categories. Instead, it only uses data of new categories of a few samples, and can be applied to anomaly detection tasks of objects of new categories.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a method and system for detecting anomalies in a small number of images based on image registration. Background Art

[0002] Currently, deep learning-based technologies have achieved remarkable results in object classification tasks. However, for anomaly detection tasks, it is too costly to collect enough abnormal data for model training. In the standard anomaly detection process, only normal data is usually provided for model training, requiring the anomaly detection method to have the ability to detect data anomalies without abnormal data training. The few-shot anomaly detection task is mainly used in anomaly detection scenarios of multi-category data. During training, only a limited number of normal images are provided for the anomaly detection task of each category. The promotion of few-shot anomaly detection can help alleviate the burden of large-scale collection of training data and reduce the data collection work for data-intensive application tasks.

[0003] Existing few-shot anomaly detection methods all follow the standard single-model single-category learning paradigm, that is, different models are trained for data of different object categories, and each model can only be used to perform anomaly detection tasks for a single object category. Existing methods need to retrain the model before performing anomaly detection tasks for unknown types of objects, which consumes a lot of time and computing resources. In fact, by designing a general few-shot anomaly detection algorithm, the anomaly detection capability of the model can be generalized to objects of unfamiliar categories. Then the algorithm will greatly reduce the computational overhead, be able to quickly perform anomaly detection for new categories of objects, and promote the application and deployment of anomaly detection in fields such as industrial production.

[0004] Patent document CN106951899A (application number: CN201710192706.6) discloses an anomaly detection method based on image recognition, including: normalizing a picture containing a detected target to obtain a grayscale image; using a trained target recognition model to perform image clipping to clip the detected target image from the grayscale image; using a trained binary classification model to perform binary classification on the detected target image to determine the credibility score of the detected target image; if the credibility score of the detected target image is not higher than a preset abnormal threshold, the detected target image is determined to be an abnormal target. By converting a picture containing a detected target into a grayscale image, the feature dimension contained in the picture can be effectively reduced without reducing the feature information of the picture; by clipping the detected target image from the grayscale image, the interference caused by the non-detection target image information can be effectively reduced. However, the invention cannot use new category data with a small number of samples to apply to the anomaly detection task of new category objects. Summary of the invention

[0005] In view of the defects in the prior art, an object of the present invention is to provide a method and system for detecting anomalies in a small number of images based on image registration.

[0006] According to the present invention, a few-sample image anomaly detection method based on image registration includes:

[0007] Step S1: extracting high-dimensional features of the supporting image and the image to be detected;

[0008] Step S2: performing spatial transformation on the high-dimensional features of the image to obtain transformed image features;

[0009] Step S3: Implement feature encoding for the transformed image features;

[0010] Step S4: performing feature registration on the coded features;

[0011] Step S5: fitting the transformed image features to the feature distribution of the supporting image to obtain a feature distribution model;

[0012] Step S6: Implement image anomaly assessment on the transformed image features and feature distribution model.

[0013] Preferably, in step S1:

[0014] Taking the support image and the image to be detected as input, a deep convolutional neural network is used to extract the high-dimensional feature information of the image. The image feature extraction network consists of three cascaded residual-based convolutional neural network modules, denoted as C1, C2 and C3, respectively, to obtain three multi-scale high-dimensional features, denoted as and

[0015] In step S2:

[0016] The high-dimensional features of the image obtained by extracting image features and Use spatial transformation neural network to perform spatial transformation of features; among them, high-dimensional features As input, i = 1, 2, 3, the spatial transformation neural network uses the following formula to transform the spatial coordinates:

[0017]

[0018] in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, S i is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ijIt is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm to obtain the transformed image features. and in, is the feature of the image to be tested, To support the features of the image.

[0019] Preferably, in step S3:

[0020] The transformed image features obtained by the feature space transformation step and Use deep convolutional neural network to realize feature encoding; among them, for the features of the image to be tested Use the encoder and predictor to get the encoded features:

[0021]

[0022] p a =P(z a )

[0023] Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and z a is the high-dimensional feature of the image to be tested after being encoded by the encoder, p a It is the high-dimensional feature of the image to be detected after being encoded by the predictor; for the feature of the supporting image Use the encoder and predictor to get the encoded features:

[0024]

[0025] p b =P(z b )

[0026] The encoder E and predictor P share weights with the encoder and predictor acting on the features of the image to be tested, z b is the high-dimensional feature of the supporting image after being encoded by the encoder, p b It is the high-dimensional feature of the image to be detected after being encoded by the predictor;

[0027] In step S4:

[0028] The obtained encoding feature p a and z b Use the image feature registration loss function to achieve feature registration:

[0029]

[0030] Among them, ||·||2 is the L-2 regularization operation;

[0031] The symmetric image feature registration loss function L is defined as:

[0032]

[0033] Preferably, in step S5:

[0034] According to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model;

[0035] The transformed features obtained by transforming the feature space and Use a statistics-based estimator to estimate the normal distribution of features, and use multivariate Gaussian distribution to obtain the probability representation of the corresponding features of normal images. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution; at each grid position (i, j), record is the transformed feature from N supporting images The collection of F ij From the multivariate Gaussian distribution N(μ ij ,∑ ij ), whose sample mean is denoted by μ ij , sample covariance∑ ij for:

[0036]

[0037] in,(·) T It is a matrix transpose operation, and the regularization term ∈I makes the sample covariance matrix full rank and reversible; the multivariate Gaussian distribution of each possible position together constitutes the feature distribution model.

[0038] Preferably, in step S6:

[0039] Based on the transformed image features obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation, an abnormality assessment function is used to implement image abnormality assessment;

[0040] For the image to be tested, the features of the image to be tested obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation are compared to calculate the following abnormality assessment function:

[0041]

[0042] in, is the sample covariance ∑ ijThe inverse matrix of the Mahalanobis distance matrix M = (M (f ij )) 1≤i≤W，1≤j≤H The anomaly score matrix is ​​formed;

[0043] Among them, the positions in the matrix where the values ​​are greater than the preset values ​​represent abnormal areas, and the abnormal score of the entire image is the maximum value of the abnormal matrix.

[0044] According to the present invention, a few-sample image anomaly detection system based on image registration is provided, comprising:

[0045] Module M1: extract high-dimensional features of the supporting image and the image to be detected;

[0046] Module M2: Perform spatial transformation on the high-dimensional features of the image to obtain transformed image features;

[0047] Module M3: Implement feature encoding for transformed image features;

[0048] Module M4: realize feature registration for coding features;

[0049] Module M5: Fitting the feature distribution of the supporting image to the transformed image features to obtain a feature distribution model;

[0050] Module M6: Implement image anomaly assessment based on transformed image features and feature distribution models.

[0051] Preferably, in the module M1:

[0052] Taking the support image and the image to be detected as input, a deep convolutional neural network is used to extract the high-dimensional feature information of the image. The image feature extraction network consists of three cascaded residual-based convolutional neural network modules, denoted as C1, C2 and C3, respectively, to obtain three multi-scale high-dimensional features, denoted as and

[0053] In the module M2:

[0054] The high-dimensional features of the image obtained by extracting image features and Use spatial transformation neural network to perform spatial transformation of features; among them, high-dimensional features As input, i = 1, 2, 3, the spatial transformation neural network uses the following formula to transform the spatial coordinates:

[0055]

[0056] in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, S i is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ij It is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm to obtain the transformed image features. and in, is the feature of the image to be tested, To support the features of the image.

[0057] Preferably, in the module M3:

[0058] The transformed image features obtained by the feature space transformation step and Use deep convolutional neural network to realize feature encoding; among them, for the features of the image to be tested Use the encoder and predictor to get the encoded features:

[0059]

[0060] p a =P(z a )

[0061] Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and z a is the high-dimensional feature of the image to be tested after being encoded by the encoder, p a It is the high-dimensional feature of the image to be detected after being encoded by the predictor; for the feature of the supporting image Use the encoder and predictor to get the encoded features:

[0062]

[0063] p b =P(z b )

[0064] The encoder E and predictor P share weights with the encoder and predictor acting on the features of the image to be tested, z b is the high-dimensional feature of the supporting image after being encoded by the encoder, p b It is the high-dimensional feature of the image to be detected after being encoded by the predictor;

[0065] In the module M4:

[0066] The obtained encoding feature p a and z bUse the image feature registration loss function to achieve feature registration:

[0067]

[0068] Among them, ||·||2 is the L-2 regularization operation;

[0069] The symmetric image feature registration loss function L is defined as:

[0070]

[0071] Preferably, in the module M5:

[0072] According to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model;

[0073] The transformed features obtained by transforming the feature space and Use a statistics-based estimator to estimate the normal distribution of features, and use multivariate Gaussian distribution to obtain the probability representation of the corresponding features of normal images. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution; at each grid position (i, j), record is the transformed feature from N supporting images The collection of F ij From the multivariate Gaussian distribution N(μ ij ,∑ ij ), whose sample mean is denoted by μ ij , sample covariance∑ ij for:

[0074]

[0075] in,(·) T It is a matrix transpose operation, and the regularization term ∈I makes the sample covariance matrix full rank and reversible; the multivariate Gaussian distribution of each possible position together constitutes the feature distribution model.

[0076] Preferably, in the module M6:

[0077] Based on the transformed image features obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation, an abnormality assessment function is used to implement image abnormality assessment;

[0078] For the image to be tested, the features of the image to be tested obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation are compared to calculate the following abnormality assessment function:

[0079]

[0080] in, is the sample covariance ∑ ij The inverse matrix of the Mahalanobis distance matrix M = (M (f ij )) 1≤i≤W，1≤j≤H The anomaly score matrix is ​​formed;

[0081] Among them, the positions in the matrix where the values ​​are greater than the preset values ​​represent abnormal areas, and the abnormal score of the entire image is the maximum value of the abnormal matrix.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] 1. Aiming at the problems existing in the current anomaly detection methods, the present invention proposes a few-sample anomaly detection method based on image registration. The present invention uses the known category object data to train a generalizable general model. It does not need to retrain the model for the new category object data. Instead, it only uses the new category data of a few samples and can be applied to the anomaly detection task of the new category object.

[0084] 2. The present invention draws on the actual human behavior of detecting anomalies and introduces training based on image registration to improve the anomaly generalization and detection ability of the anomaly detection algorithm, greatly reducing the data collection and high-performance computing overhead required for anomaly detection task model training, thereby achieving better performance in anomaly detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0086] Figure 1 A flow chart of a method in one embodiment of the present invention;

[0087] Figure 2 Schematic diagram of a system in one embodiment of the present invention. DETAILED DESCRIPTION

[0088] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0089] Embodiment 1:

[0090] The present invention provides a few-sample image anomaly detection method based on image registration, comprising: an image feature extraction step: for a support image and an image to be detected, a deep convolutional neural network is used to extract high-dimensional features of the image; a feature space transformation step: for the high-dimensional features of the image obtained in the image feature extraction step, a space transformation neural network is used to perform feature space transformation of the features to obtain transformed image features; a feature encoding step: for the transformed image features obtained in the feature space transformation step, a deep convolutional neural network is used to implement feature encoding; a feature registration step: for the encoded features obtained in the feature encoding step, an image feature registration loss function is used to implement feature registration; a feature distribution estimation step: according to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model; an anomaly assessment step: according to the transformed image features obtained in the feature space transformation step and the feature distribution model obtained in the feature distribution estimation step, an anomaly assessment function is used to implement image anomaly assessment. The present invention draws on the actual human behavior of detecting anomalies and introduces training based on image registration to improve the anomaly generalization and detection ability of the anomaly detection algorithm, greatly reducing the data collection and high-performance computing overhead required for anomaly detection task model training, thereby achieving better performance in anomaly detection tasks.

[0091] According to a few-sample image anomaly detection method based on image registration provided by the present invention, Figure 1-Figure 2 As shown, including:

[0092] Step S1: extracting high-dimensional features of the supporting image and the image to be detected;

[0093] Specifically, in step S1:

[0094] Taking the support image and the image to be detected as input, a deep convolutional neural network is used to extract the high-dimensional feature information of the image. The image feature extraction network consists of three cascaded residual-based convolutional neural network modules, denoted as C1, C2 and C3, respectively, to obtain three multi-scale high-dimensional features, denoted as and

[0095] Step S2: performing spatial transformation on the high-dimensional features of the image to obtain transformed image features;

[0096] In step S2:

[0097] The high-dimensional features of the image obtained by extracting image features and Use spatial transformation neural network to perform spatial transformation of features; among them, high-dimensional features As input, i = 1, 2, 3, the spatial transformation neural network uses the following formula to transform the spatial coordinates:

[0098]

[0099] in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, S i is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ij It is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm to obtain the transformed image features. and in, is the feature of the image to be tested, To support the features of the image.

[0100] Step S3: Implement feature encoding for the transformed image features;

[0101] Specifically, in step S3:

[0102] The transformed image features obtained by the feature space transformation step and Use deep convolutional neural network to realize feature encoding; among them, for the features of the image to be tested Use the encoder and predictor to get the encoded features:

[0103]

[0104] p a =P(z a )

[0105] Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and z a is the high-dimensional feature of the image to be tested after being encoded by the encoder, p a It is the high-dimensional feature of the image to be detected after being encoded by the predictor; for the feature of the supporting image Use the encoder and predictor to get the encoded features:

[0106]

[0107] p b =P(z b )

[0108] The encoder E and predictor P share weights with the encoder and predictor acting on the features of the image to be tested, z b is the high-dimensional feature of the supporting image after being encoded by the encoder, p b It is the high-dimensional feature of the image to be detected after being encoded by the predictor;

[0109] Step S4: performing feature registration on the coded features;

[0110] In step S4:

[0111] The obtained encoding feature p a and z b Use the image feature registration loss function to achieve feature registration:

[0112]

[0113] Among them, ||·||2 is the L-2 regularization operation;

[0114] The symmetric image feature registration loss function L is defined as:

[0115]

[0116] Step S5: fitting the transformed image features to the feature distribution of the supporting image to obtain a feature distribution model;

[0117] Specifically, in step S5:

[0118] According to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model;

[0119] The transformed features obtained by transforming the feature space and Use a statistics-based estimator to estimate the normal distribution of features, and use multivariate Gaussian distribution to obtain the probability representation of the corresponding features of normal images. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution; at each grid position (i, j), record is the transformed feature from N supporting images The collection of F ij From the multivariate Gaussian distribution N(μ ij ,∑ ij ), whose sample mean is denoted by μ ij , sample covariance∑ ij for:

[0120]

[0121] in,(·) T It is a matrix transpose operation, and the regularization term ∈I makes the sample covariance matrix full rank and reversible; the multivariate Gaussian distribution of each possible position together constitutes the feature distribution model.

[0122] Step S6: Implement image anomaly assessment on the transformed image features and feature distribution model.

[0123] Specifically, in step S6:

[0124] Based on the transformed image features obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation, an abnormality assessment function is used to implement image abnormality assessment;

[0125] For the image to be tested, the features of the image to be tested obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation are compared to calculate the following abnormality assessment function:

[0126]

[0127] in, is the sample covariance ∑ ij The inverse matrix of the Mahalanobis distance matrix M = (M (f ij )) 1≤i≤W，1≤j≤H The anomaly score matrix is ​​formed;

[0128] Among them, the positions in the matrix where the values ​​are greater than the preset values ​​represent abnormal areas, and the abnormal score of the entire image is the maximum value of the abnormal matrix.

[0129] Embodiment 2:

[0130] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0131] Those skilled in the art may understand the image registration-based few-sample image anomaly detection method provided by the present invention as a specific implementation of the image registration-based few-sample image anomaly detection system, that is, the image registration-based few-sample image anomaly detection system may be implemented by executing the step flow of the image registration-based few-sample image anomaly detection method.

[0132] According to the present invention, a few-sample image anomaly detection system based on image registration is provided, comprising:

[0133] Module M1: extract high-dimensional features of the supporting image and the image to be detected;

[0134] Specifically, in the module M1:

[0135] Taking the support image and the image to be detected as input, a deep convolutional neural network is used to extract the high-dimensional feature information of the image. The image feature extraction network consists of three cascaded residual-based convolutional neural network modules, denoted as C1, C2 and C3, respectively, to obtain three multi-scale high-dimensional features, denoted as and

[0136] Module M2: Perform spatial transformation on the high-dimensional features of the image to obtain transformed image features;

[0137] In the module M2:

[0138] The high-dimensional features of the image obtained by extracting image features and Use spatial transformation neural network to perform spatial transformation of features; among them, high-dimensional features As input, i = 1, 2, 3, the spatial transformation neural network uses the following formula to transform the spatial coordinates:

[0139]

[0140] in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, S i is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ij It is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm to obtain the transformed image features. and in, is the feature of the image to be tested, To support the features of the image.

[0141] Module M3: Implement feature encoding for transformed image features;

[0142] Specifically, in the module M3:

[0143] The transformed image features obtained by the feature space transformation step and Use deep convolutional neural network to realize feature encoding; among them, for the features of the image to be tested Use the encoder and predictor to get the encoded features:

[0144]

[0145] p a =P(z a )

[0146] Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and z a is the high-dimensional feature of the image to be tested after being encoded by the encoder, p a It is the high-dimensional feature of the image to be detected after being encoded by the predictor; for the feature of the supporting image Use the encoder and predictor to get the encoded features:

[0147]

[0148] p b =P(z b )

[0149] The encoder E and predictor P share weights with the encoder and predictor acting on the features of the image to be tested, z b is the high-dimensional feature of the supporting image after being encoded by the encoder, p b It is the high-dimensional feature of the image to be detected after being encoded by the predictor;

[0150] Module M4: realize feature registration for coding features;

[0151] In the module M4:

[0152] The obtained encoding feature p a and z b Use the image feature registration loss function to achieve feature registration:

[0153]

[0154] Among them, ||·||2 is the L-2 regularization operation;

[0155] The symmetric image feature registration loss function L is defined as:

[0156]

[0157] Module M5: Fitting the feature distribution of the supporting image to the transformed image features to obtain a feature distribution model;

[0158] Specifically, in the module M5:

[0159] According to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model;

[0160] The transformed features obtained by transforming the feature space and Use a statistics-based estimator to estimate the normal distribution of features, and use multivariate Gaussian distribution to obtain the probability representation of the corresponding features of normal images. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution; at each grid position (i, j), record is the transformed feature from N supporting images The collection of F ij From the multivariate Gaussian distribution N(μ ij ,∑ ij ), whose sample mean is denoted by μ ij , sample covariance∑ ij for:

[0161]

[0162] in,(·) T It is a matrix transpose operation, and the regularization term ∈I makes the sample covariance matrix full rank and reversible; the multivariate Gaussian distribution of each possible position together constitutes the feature distribution model.

[0163] Module M6: Implement image anomaly assessment based on transformed image features and feature distribution models.

[0164] Specifically, in the module M6:

[0165] Based on the transformed image features obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation, an abnormality assessment function is used to implement image abnormality assessment;

[0166] For the image to be tested, the features of the image to be tested obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation are compared to calculate the following abnormality assessment function:

[0167]

[0168] in, is the sample covariance ∑ ij The inverse matrix of the Mahalanobis distance matrix M = (M (f ij )) 1≤i≤W，1≤j≤H The anomaly score matrix is ​​formed;

[0169] Among them, the positions in the matrix where the values ​​are greater than the preset values ​​represent abnormal areas, and the abnormal score of the entire image is the maximum value of the abnormal matrix.

[0170] Embodiment 3:

[0171] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0172] like Figure 1 As shown, it is a flow chart of an embodiment of a few-sample image anomaly detection method based on image registration of the present invention, wherein the method uses a deep convolutional neural network to extract high-dimensional features of the image for the supporting image and the image to be detected; uses a spatial transformation neural network to perform spatial transformation of the high-dimensional features of the image obtained in the image feature extraction step to obtain transformed image features; uses a deep convolutional neural network to implement feature encoding for the transformed image features obtained in the feature space transformation step; implements feature registration for the encoded features obtained in the feature encoding step using an image feature registration loss function; uses a distribution estimation model to fit the feature distribution of the supporting image according to the transformed image features obtained in the feature space transformation step to obtain a feature distribution model; uses an anomaly assessment function to implement image anomaly assessment according to the transformed image features obtained in the feature space transformation step and the feature distribution model obtained in the feature distribution estimation step.

[0173] In view of the problems existing in the current anomaly detection methods, the present invention proposes a few-sample anomaly detection method based on image registration. The method uses data of known categories of objects to train a generalizable model. It does not need to retrain the model for data of new categories of objects. Instead, it can be applied to the anomaly detection task of new categories of objects by using only a few samples of new category data. By drawing on the actual behavior of human beings in detecting anomalies and introducing training based on image registration, the present invention improves the anomaly generalization and detection ability of the anomaly detection algorithm, greatly reduces the data collection and high-performance computing overhead required for training the anomaly detection task model, and thus achieves better performance in the anomaly detection task.

[0174] Specifically, refer to Figure 1 , the method comprises the following steps:

[0175] Image feature extraction step: For the support image and the image to be detected, a deep convolutional neural network is used to extract the high-dimensional features of the image;

[0176] Feature space transformation step: using a spatial transformation neural network to perform spatial transformation of the high-dimensional features of the image obtained in the image feature extraction step to obtain transformed image features;

[0177] Feature encoding step: using a deep convolutional neural network to implement feature encoding for the transformed image features obtained in the feature space transformation step;

[0178] Feature registration step: using an image feature registration loss function to realize feature registration for the encoded features obtained in the feature encoding step;

[0179] Feature distribution estimation step: according to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model;

[0180] Abnormality assessment step: Based on the transformed image features obtained in the feature space transformation step and the feature distribution model obtained in the feature distribution estimation step, an abnormality assessment function is used to implement image abnormality assessment.

[0181] Corresponding to the above method, the present invention also provides an embodiment of a few-sample image anomaly detection system based on image registration, comprising:

[0182] Image feature extraction module: for the support image and the image to be detected, a deep convolutional neural network is used to extract the high-dimensional features of the image;

[0183] Feature space transformation module: for high-dimensional features of the image obtained by the image feature extraction module, a spatial transformation neural network is used to perform spatial transformation of the features to obtain transformed image features;

[0184] Feature encoding module: using a deep convolutional neural network to implement feature encoding for the transformed image features obtained by the feature space transformation module;

[0185] Feature registration module: for the coded features obtained by the feature coding module, the image feature registration loss function is used to realize feature registration;

[0186] Feature distribution estimation module: according to the transformed image features obtained by the feature space transformation module, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model;

[0187] Abnormality assessment module: Based on the transformed image features obtained by the feature space transformation module and the feature distribution model obtained by the feature distribution estimation module, an abnormality assessment function is used to implement image abnormality assessment.

[0188] The technical features implemented by each module of the above-mentioned image registration-based few-sample image anomaly detection system may be the same as the technical features implemented by the corresponding steps in the above-mentioned image registration-based few-sample image anomaly detection method.

[0189] The specific implementation of each of the above steps and modules is described in detail below to facilitate understanding of the technical solution of the present invention.

[0190] In some embodiments of the present invention, the image feature extraction step includes: extracting high-dimensional features of the image using a deep convolutional neural network for the support image and the image to be detected;

[0191] In some embodiments of the present invention, the feature space transformation step includes: using a spatial transformation neural network to perform spatial transformation of the high-dimensional features of the image obtained in the image feature extraction step to obtain transformed image features;

[0192] In some embodiments of the present invention, the feature encoding step includes: using a deep convolutional neural network to implement feature encoding for the transformed image features obtained in the feature space transformation step;

[0193] In some embodiments of the present invention, the feature registration step, wherein: the encoded features obtained in the feature encoding step are subjected to feature registration using an image feature registration loss function;

[0194] In some embodiments of the present invention, the feature distribution estimation step includes: fitting the feature distribution of the support image using a distribution estimation model according to the transformed image features obtained in the feature space transformation step to obtain a feature distribution model;

[0195] In some embodiments of the present invention, the abnormality assessment step includes: implementing image abnormality assessment using an abnormality assessment function based on the transformed image features obtained in the feature space transformation step and the feature distribution model obtained in the feature distribution estimation step.

[0196] Specifically, the training system network framework consisting of an image feature extraction module, a feature space transformation module, a feature encoding module, a feature registration module, a feature distribution estimation module, and an anomaly assessment module is as follows: Figure 2 As shown, the entire system framework can be trained end-to-end.

[0197] In such Figure 2 In the system framework of the embodiment shown, the support image (i.e., a small number of normal images determined) and the image to be detected are used as input, and a deep convolutional neural network is used to extract high-dimensional feature information of the image. The image feature extraction network is composed of three cascaded residual-based convolutional neural network modules, which are respectively denoted as C1, C2 and C3, and three multi-scale high-dimensional features are obtained, which are respectively denoted as and

[0198] In such Figure 2 In the system framework of the embodiment shown in the figure, the high-dimensional features of the image obtained in the image feature extraction step are and Use spatial transformation neural network to perform spatial transformation of features. As input, the spatial transformation neural network uses the following formula to transform the spatial coordinates:

[0199]

[0200] in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, Si is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ij It is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm, and finally obtains the transformed image features. (features of the image to be tested) and (Features of the supporting image).

[0201] In such Figure 2 In the system framework of the embodiment shown in the figure, the transformed image features obtained by the feature space transformation step are (features of the image to be tested) and (Features of the supporting image), use deep convolutional neural network to implement feature encoding. Among them, for the features of the image to be tested Use encoder and predictor to get encoded features,

[0202]

[0203] Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and p a It is the encoded high-dimensional feature of the image to be detected. Use only the encoder to get the encoded features,

[0204]

[0205] The encoder E shares weights with the encoder acting on the features of the image to be tested, z b It is the encoded high-dimensional feature of the supporting image.

[0206] In such Figure 2 In the system framework of the embodiment shown in the figure, the encoding feature p obtained in the feature encoding step is a and z b , feature registration is achieved using the image feature registration loss function,

[0207]

[0208] Among them, ||.||2 is the L-2 regularization operation. Finally, the symmetric image feature registration loss function is defined as,

[0209]

[0210] In such Figure 2 In the system framework of the embodiment shown, the transformed features obtained by the feature space transformation step are and A statistically based estimator is used to estimate the normal distribution of features, and a multivariate Gaussian distribution is used to obtain the probability representation of the corresponding features of the normal image. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution. At each grid position (i, j), record is the transformed feature from N supporting images Assume that F ij From the multivariate Gaussian distribution N(μ ij ,∑ ij ), whose sample mean is denoted by μ ij , sample covariance∑ ij for:

[0211]

[0212] Among them, the regularization term ∈I makes the sample covariance matrix full rank and invertible. The multivariate Gaussian distribution of each possible position together constitutes the final feature distribution model.

[0213] In such Figure 2 In the system framework of the embodiment shown, for the image to be tested, the features of the image to be tested obtained in the feature space transformation step and the feature distribution model obtained in the feature distribution estimation step are compared to calculate the following abnormality assessment function:

[0214]

[0215] The above Mahalanobis distance matrix M=(M(f ij )) 1≤i≤W，1≤j≤H The final anomaly score matrix is ​​formed. Among them, the position with a larger value in the matrix represents the abnormal area. The final anomaly score of the entire image is the maximum value of the above anomaly matrix.

[0216] In summary, the present invention aims at the problems existing in the current anomaly detection methods and proposes a few-sample anomaly detection method based on image registration. This method uses data of known categories of objects to train a generalizable model. It does not need to retrain the model for data of new categories of objects. Instead, it can be applied to the anomaly detection task of new categories of objects by using only a few samples of new category data. By drawing on the actual behavior of humans in detecting anomalies and introducing training based on image registration, the present invention improves the anomaly generalization and detection ability of the anomaly detection algorithm, greatly reduces the data collection and high-performance computing overhead required for training the anomaly detection task model, and thus achieves better performance in the anomaly detection task.

[0217] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0218] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A few-sample image anomaly detection method based on image registration, characterized in that: include: Step S1: extracting high-dimensional features of the supporting image and the image to be detected; Step S2: performing spatial transformation on the high-dimensional features of the image to obtain transformed image features; Step S3: Implement feature encoding for the transformed image features; Step S4: performing feature registration on the coded features; Step S5: fitting the transformed image features to the feature distribution of the supporting image to obtain a feature distribution model; Step S6: Implementing image anomaly assessment on the transformed image features and feature distribution model; In step S5: According to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model; The transformed features obtained by transforming the feature space and Use a statistics-based estimator to estimate the normal distribution of features, and use multivariate Gaussian distribution to obtain the probability representation of the corresponding features of normal images. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution; at each grid position (i, j), record is the transformed feature from N supporting images The collection of F ij From the multivariate Gaussian distribution N(μ ij ,Σ ij ), whose sample mean is denoted by μ ij , sample covariance Σ ij for: in,(·) T is a matrix transposition operation, and the regularization term ∈I makes the sample covariance matrix full rank and reversible; the multivariate Gaussian distribution of each possible position together constitutes the feature distribution model; In step S6: Based on the transformed image features obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation, an abnormality assessment function is used to implement image abnormality assessment; For the image to be tested, the features of the image to be tested obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation are compared to calculate the following abnormality assessment function: in, is the sample covariance Σ ij The inverse matrix of the Mahalanobis distance matrix M = (M (f ij )) 1≤i≤W,1≤j≤H The anomaly score matrix is ​​formed; Among them, the positions in the matrix where the values ​​are greater than the preset values ​​represent abnormal areas, and the abnormal score of the entire image is the maximum value of the abnormal matrix.

2. The method for detecting anomalies in a few-sample image based on image registration according to claim 1, characterized in that: In step S1: Taking the support image and the image to be detected as input, a deep convolutional neural network is used to extract the high-dimensional feature information of the image. The image feature extraction network consists of three cascaded residual-based convolutional neural network modules, denoted as C1, C2 and C3, respectively, to obtain three multi-scale high-dimensional features, denoted as and In step S2: The high-dimensional features of the image obtained by extracting image features and Use spatial transformation neural network to perform spatial transformation of features; among them, high-dimensional features As input, i = 1, 2, 3, the spatial transformation neural network uses the following formula to transform the spatial coordinates: in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, S i is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ij It is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm to obtain the transformed image features. and in, is the feature of the image to be tested, To support the features of the image.

3. The method for detecting anomalies in a few-sample images based on image registration according to claim 1, characterized in that: In step S3: The transformed image features obtained by the feature space transformation step and Use deep convolutional neural network to realize feature encoding; among them, for the features of the image to be tested Use the encoder and predictor to get the encoded features: p a =P(z a ) Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and z a is the high-dimensional feature of the image to be tested after being encoded by the encoder, p a It is the high-dimensional feature of the image to be detected after being encoded by the predictor; for the feature of the supporting image Use the encoder and predictor to get the encoded features: p b =P(z b ) The encoder E and predictor P share weights with the encoder and predictor acting on the features of the image to be tested, z b is the high-dimensional feature of the supporting image after being encoded by the encoder, p b It is the high-dimensional feature of the image to be detected after being encoded by the predictor; In step S4: The obtained encoding feature p a and z b Use the image feature registration loss function to achieve feature registration: Among them, ||·||2 is the L-2 regularization operation; The symmetric image feature registration loss function L is defined as:

4. A few-sample image anomaly detection system based on image registration, characterized in that: include: Module M1: extract high-dimensional features of the supporting image and the image to be detected; Module M2: Perform spatial transformation on the high-dimensional features of the image to obtain transformed image features; Module M3: Implement feature encoding for transformed image features; Module M4: realize feature registration for coding features; Module M5: Fitting the feature distribution of the supporting image to the transformed image features to obtain a feature distribution model; Module M6: Implement image anomaly assessment based on transformed image features and feature distribution models; In the module M5: According to the transformed image features obtained in the feature space transformation step, a distribution estimation model is used to fit the feature distribution of the support image to obtain a feature distribution model; The transformed features obtained by transforming the feature space and Use a statistics-based estimator to estimate the normal distribution of features, and use multivariate Gaussian distribution to obtain the probability representation of the corresponding features of normal images. Assume that the image is divided into a grid (i, j) ∈ [1, W] × [1, H], where W × H is the resolution of the features used to estimate the normal distribution; at each grid position (i, j), record is the transformed feature from N supporting images The collection of F ij From the multivariate Gaussian distribution N(μ ij ,Σ ij ), whose sample mean is denoted by μ ij , sample covariance Σ ij for: in,(·) T is a matrix transposition operation, and the regularization term ∈I makes the sample covariance matrix full rank and reversible; the multivariate Gaussian distribution of each possible position together constitutes the feature distribution model; In the module M6: Based on the transformed image features obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation, an abnormality assessment function is used to implement image abnormality assessment; For the image to be tested, the features of the image to be tested obtained by feature space transformation and the feature distribution model obtained by feature distribution estimation are compared to calculate the following abnormality assessment function: in, is the sample covariance Σ ij The inverse matrix of the Mahalanobis distance matrix M = (M (f ij )) 1≤i≤W,1≤j≤H The anomaly score matrix is ​​formed; Among them, the positions in the matrix where the values ​​are greater than the preset values ​​represent abnormal areas, and the abnormal score of the entire image is the maximum value of the abnormal matrix.

5. The few-sample image anomaly detection system based on image registration according to claim 4, characterized in that: In the module M1: Taking the support image and the image to be detected as input, a deep convolutional neural network is used to extract the high-dimensional feature information of the image. The image feature extraction network consists of three cascaded residual-based convolutional neural network modules, denoted as C1, C2 and C3, respectively, to obtain three multi-scale high-dimensional features, denoted as and In the module M2: The high-dimensional features of the image obtained by extracting image features and Use spatial transformation neural network to perform spatial transformation of features; among them, high-dimensional features As input, i = 1, 2, 3, the spatial transformation neural network uses the following formula to transform the spatial coordinates: in, is the input feature before transformation The original coordinates of is the transformed output feature The target coordinates, S i is the i-th spatial transformation neural network, A i is the coordinate transformation matrix, θ ij It is the specific parameter of the coordinate affine transformation matrix. The spatial transformation neural network uses the convolutional neural network to continuously correct the parameters of the affine transformation matrix according to the error back propagation algorithm to obtain the transformed image features. and in, is the feature of the image to be tested, To support the features of the image.

6. The few-sample image anomaly detection system based on image registration according to claim 4, characterized in that: In the module M3: The transformed image features obtained by the feature space transformation step and Use deep convolutional neural network to realize feature encoding; among them, for the features of the image to be tested Use the encoder and predictor to get the encoded features: p a =P(z a ) Among them, E is an encoder composed of three layers of convolution operations, P is a predictor composed of two layers of convolution operations, and z a is the high-dimensional feature of the image to be tested after being encoded by the encoder, p a It is the high-dimensional feature of the image to be detected after being encoded by the predictor; for the feature of the supporting image Use the encoder and predictor to get the encoded features: p b =P(z b ) The encoder E and predictor P share weights with the encoder and predictor acting on the features of the image to be tested, z b is the high-dimensional feature of the supporting image after being encoded by the encoder, p b It is the high-dimensional feature of the image to be detected after being encoded by the predictor; In the module M4: The obtained encoding feature p a and z b Use the image feature registration loss function to achieve feature registration: Among them, ||·||2 is the L-2 regularization operation; The symmetric image feature registration loss function L is defined as:

Images