Model training and image detection method and device, electronic equipment, storage medium and computer program product

Abstract

The invention discloses a model training and image detection method and device, electronic equipment, a storage medium and a computer program product. The method comprises the steps that a first image set is utilized to generate a second image set, the first image set comprises one or more first images, each first image comprises one or more normal areas, the second image set comprises one or more second images, and each second image comprises one or more abnormal areas; generating a first loss function by using the first image set and the second image set, the first loss function at least being capable of constraining a model training direction associated with an abnormal region; and training the first model by using the first loss function and a second loss function based on a knowledge distillation algorithm, the second loss function at least being capable of constraining a model training direction associated with a normal region, and the first model being used for performing anomaly detection on a to-be-detected image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a model training and image detection method and device, electronic equipment, storage medium and computer program product. BACKGROUND

[0002] Image anomaly detection technology is to identify abnormal images and accurately locate the specific area (also known as an abnormal area) deviating from the normal mode. This technology is widely used in many fields, such as industrial quality inspection, medical diagnosis, or video monitoring, etc. However, image anomaly detection technology usually faces problems caused by the scarcity of abnormal samples and the wide variety of abnormal types. In order to solve the above problems, an unsupervised anomaly detection scheme has been proposed, which can minimize the need for a large amount of labeled data.

[0003] In related unsupervised anomaly detection schemes, one way is to establish a corresponding relationship between a student network and a teacher network based on a knowledge distillation method, and to realize image anomaly detection by using the inconsistency between the features of the student network and the features of the teacher network. However, in the above scheme, there may be a case of missing detection of abnormal areas, resulting in low accuracy of image anomaly detection. SUMMARY

[0004] To solve the related technical problems, the embodiments of the present application provide a model training and image detection method and device, electronic equipment, storage medium and computer program product.

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a model training method, comprising:

[0007] Generating a second image set using a first image set, the first image set containing one or more first images, each first image containing one or more normal areas, the second image set containing one or more second images, each second image containing one or more abnormal areas;

[0008] Generating a first loss function using the first image set and the second image set, the first loss function being capable of at least constraining the model training direction associated with the abnormal area;

[0009] Training the first model based on a knowledge distillation algorithm using the first loss function and a second loss function, the second loss function being capable of at least constraining the model training direction associated with the normal area, the first model being used for anomaly detection of a to-be-detected image.

[0010] In the scheme, the first loss function is generated by using the first image set and the second image set, and the first loss function comprises:

[0011] A second model is obtained, and the second model can at least guide the first model to perform model training.

[0012] A first feature vector associated with the normal region of the first image set and a second feature vector associated with the abnormal region of the second image set in the second model are determined.

[0013] A third feature vector associated with the abnormal region of the second image set in the first model is determined.

[0014] The first loss function is generated by using the first feature vector, the second feature vector and the third feature vector.

[0015] In the scheme, the first loss function is generated by using the first feature vector, the second feature vector and the third feature vector, and the first loss function comprises:

[0016] A first value is obtained by using the first feature vector and the second feature vector, and the first value represents a projection of the second feature vector on the first feature vector.

[0017] A second value is obtained by using the first feature vector and the third feature vector, and the second value represents a projection of the third feature vector on the first feature vector.

[0018] The first loss function is generated by using the first value and the second value.

[0019] In the scheme, the first loss function is generated by using the first value and the second value, and the first loss function comprises:

[0020] A third value is obtained by using the second feature vector and the third feature vector, and the third value represents a difference degree between the second feature vector and the third feature vector.

[0021] The first loss function is generated by using the first value, the second value and the third value.

[0022] In the scheme, in the model training process, the distance between the second feature vector and the third feature vector is increased by using the first loss function.

[0023] In the scheme, when the first value is positive, a mirror direction of the second feature vector relative to the first feature vector is taken as an adjustment direction, and the distance between the third feature vector and the second feature vector is increased.

[0024] In the above scheme, in the case that the first value is negative, the opposite direction of the second feature vector relative to the first feature vector is taken as the adjustment direction, and the distance between the third feature vector and the second feature vector is increased.

[0025] In the above scheme, the generating the second image set by using the first image set comprises:

[0026] An image set is obtained, the third image set comprising one or more third images, each third image comprising an abnormal region.

[0027] A first image is selected from the first image set, and one or more third images are selected from the third image set.

[0028] The selected one or more third images and the first image are fused to obtain a second image.

[0029] In the above scheme, the fusing the selected one or more third images and the first image to obtain a second image comprises:

[0030] A target position of each third image in the selected one or more third images relative to the first image is determined.

[0031] The selected one or more third images and the first image are fused by using the determined target position to obtain a second image.

[0032] Embodiments of the present application also provide an image detection method, comprising:

[0033] An image to be detected is obtained.

[0034] The first model and the second model are used to detect the image to be detected respectively to obtain a first detection result and a second detection result, the first model being obtained according to any one of the above model training methods, the first detection result representing a detection result of the first model on the image to be detected, and the second detection result representing a detection result of the second model on the image to be detected.

[0035] The first detection result and the second detection result are used to determine whether the image to be detected is an abnormal image.

[0036] Embodiments of the present application also provide a model training device, comprising:

[0037] The first generating unit is configured to generate a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, the second image set comprising one or more second images, each second image comprising one or more abnormal regions.

[0038] The second generating unit is configured to generate a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region.

[0039] The training unit is configured to train the first model by using the first loss function and a second loss function based on a knowledge distillation algorithm, the second loss function being capable of at least constraining a model training direction associated with a normal region, the first model being used for performing abnormal detection on a to-be-detected image.

[0040] Embodiments of the present application further provide an image detection device, comprising:

[0041] The acquisition unit is configured to acquire a to-be-detected image.

[0042] The detection unit is configured to perform detection on the to-be-detected image by using a first model and a second model respectively, to obtain a first detection result and a second detection result, the first model being obtained according to any one of the model training methods, the first detection result representing a detection result of the first model on the to-be-detected image, and the second detection result representing a detection result of the second model on the to-be-detected image.

[0043] The determination unit is configured to determine whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

[0044] Embodiments of the present application further provide an electronic device, comprising a processor and a communication interface; wherein

[0045] The processor is configured to generate a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, the second image set comprising one or more second images, each second image comprising one or more abnormal regions; generate a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region; and train a first model by using the first loss function and a second loss function based on a knowledge distillation algorithm, the second loss function being capable of at least constraining a model training direction associated with a normal region, the first model being used for performing abnormal detection on a to-be-detected image.

[0046] Alternatively,

[0047] the processor is configured to: acquire, through a communication interface, a to-be-detected image; detect the to-be-detected image by using a first model and a second model respectively to obtain a first detection result and a second detection result, the first model being obtained according to any one of the model training methods, the first detection result representing a detection result of the first model on the to-be-detected image, and the second detection result representing a detection result of the second model on the to-be-detected image; and determine whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

[0048] Embodiments of the present application further provide an electronic device, comprising a processor and a memory for storing a computer program capable of running on the processor,

[0049] wherein the processor is configured to implement the steps of any one of the model training methods or the steps of any one of the image detection methods when running the computer program.

[0050] Embodiments of the present application further provide a storage medium having a computer program stored thereon, the computer program being configured to implement the steps of any one of the model training methods or the steps of any one of the image detection methods when executed by a processor.

[0051] Embodiments of the present application further provide a computer program product comprising a computer program, the computer program being configured to implement the steps of any one of the model training methods or the steps of any one of the image detection methods when executed by a processor.

[0052] The model training and image detection method, device, electronic device, storage medium and computer program product provided in the embodiments of the present application, wherein the model training method comprises: generating a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, and the second image set comprising one or more second images, each second image comprising one or more abnormal regions; generating a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region; training the first model based on a knowledge distillation algorithm by using the first loss function and a second loss function, the second loss function being capable of at least constraining a model training direction associated with a normal region, and the first model being used for abnormal detection of a to-be-detected image. The technical solution provided in the embodiments of the present application, in the process of training a student model by using a knowledge distillation algorithm, determines an optimization direction of model training based on the synthesized abnormal image and normal image, and trains the student model based on the determined optimization direction. In this way, the response difference of the student model and a teacher model in the knowledge distillation algorithm to the abnormal image can be strengthened, so that the student model and the teacher model after training can more accurately detect the abnormal region, thereby improving the accuracy of image abnormal detection. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A flowchart of a model training method according to an embodiment of the present application is shown in FIG. 1.

[0054] Figure 2 A flowchart of an image detection method according to an embodiment of the present application is shown in FIG. 2.

[0055] Figure 3 A flowchart of an image abnormal detection method according to an embodiment of the present application is shown in FIG. 3.

[0056] Figure 4 A structural diagram of a model training device according to an embodiment of the present application is shown in FIG. 4.

[0057] Figure 5 A structural diagram of an image detection device according to an embodiment of the present application is shown in FIG. 5.

[0058] Figure 6 A structural diagram of an electronic device according to an embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0059] The present application will be described in further detail below with reference to the accompanying drawings and embodiments.

[0060] In the related art, an unsupervised anomaly detection scheme based on knowledge distillation includes: in a training phase, a teacher model (also understood as a teacher network) is pre-trained using an image classification dataset (such as ImageNet), and a student model (also understood as a student network) is trained using normal images in an anomaly detection task; in a test phase, the student model can usually mimic the feature distribution of the teacher model in the normal feature area, and there is a difference (i.e., there is a different feature distribution) in the abnormal area, so that the difference can be used to identify abnormal images; wherein, multi-scale features (such as STPM and MKD schemes) of different network layers can be introduced in the knowledge distillation process to improve the overall performance of the student model and the teacher model.

[0061] However, in the case of training the model based on the knowledge distillation method, the problem of abnormal generalization often occurs, that is, the student model produces similar abnormal feature distribution as the teacher model due to its own generalization ability, which leads to missed detection of the abnormal area, so that the accuracy of image anomaly detection is not high.

[0062] Based on this, in various embodiments of the present application, in the process of training the model based on the knowledge distillation method, the direction of model optimization is dynamically adjusted by fusing normal images and simulated abnormal images to strengthen the response difference of the teacher model and the student model to the abnormal area, thereby solving the problem of abnormal generalization, enhancing the accuracy of anomaly detection, and improving the stability of model learning.

[0063] The embodiments of the present application provide a model training method, as shown in Figure 1 The method is applied to an electronic device and includes the following steps:

[0064] Step 101: generating a second image set using a first image set, the first image set containing one or more first images, each first image containing one or more normal areas, and the second image set containing one or more second images, each second image containing one or more abnormal areas;

[0065] Step 102: generating a first loss function using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal area;

[0066] Step 103: training the first model based on a knowledge distillation algorithm using the first loss function and a second loss function, the second loss function being capable of at least constraining a model training direction associated with a normal area, and the first model being used for anomaly detection of a to-be-detected image.

[0067] In actual application, the electronic device can include a device capable of performing steps 101 to 103, such as a server, a notebook computer, a tablet computer, or a desktop computer, and the like, and the embodiments of the present application do not limit this. In addition, the first image set can be referred to as a normal image set, and correspondingly, the first image can be referred to as a normal image, that is, the first image does not contain an abnormal area (which can also be understood as an image defect); the second image set can be referred to as an abnormal image set or a simulated abnormal image set, and correspondingly, the second image can be referred to as an abnormal image or a simulated abnormal image, and the embodiments of the present application do not limit the names of the first image set and the second image set.

[0068] In actual application, for the second image set, the electronic device can synthesize the second image set by using a pre-configured image set and the first image set.

[0069] Specifically, in an embodiment, the specific implementation of step 101 can include:

[0070] Obtaining a third image set containing one or more third images, each third image containing an abnormal area;

[0071] Selecting one first image from the first image set and one or more third images from the third image set;

[0072] Fusing the selected one or more third images and the first image to obtain a second image.

[0073] In actual application, the third image set can be understood as a pre-configured image set, and the third image set is referred to as an external texture data set or a texture image data set, and correspondingly, the third image can be referred to as a texture image, and the third image contains an abnormal area type containing image texture, and the embodiments of the present application do not limit the name of the third image set.

[0074] In actual application, the electronic device can obtain the third image set from a related database, and randomly select one first image and one or more third images, and sequentially superimpose each third image in the one or more third images on the selected first image to realize the fusion processing of the images.

[0075] Here, before the fusion processing of the images, the electronic device can determine an abnormal mask corresponding to each third image, so as to superimpose the third image on a specific area of the first image based on the abnormal mask.

[0076] Specifically, in an embodiment, the fusing the selected one or more third images and the first image to obtain a second image includes:

[0077] determine a target position of each of the selected one or more third images relative to the first image;

[0078] fuse the selected one or more third images and the first image by using the determined target position to obtain a second image.

[0079] In actual application, for each third image, the electronic device can generate noise information (such as two-dimensional Perlin noise) in a random manner; the noise information is binarized by using a preset first threshold, so as to obtain an abnormal mask of the third image, the abnormal mask being used to indicate a target position (which can also be understood as an abnormal region) of the third image relative to the first image; wherein the first threshold can be set as needed, such as 0.5, which is not limited in the embodiments of the present application.

[0080] Then, the third image is sequentially superimposed on the first image by using the randomly selected control factor and the determined target position, so as to synthesize a second image; wherein the following formula can be used to synthesize the second image:

[0081]

[0082] wherein, represents the second image, x n represents the first image, z represents the third image, m represents the abnormal mask, and γ represents the control factor, the value range of which is [0.15, 1].

[0083] Exemplarily, assuming that one third image and one first image are selected, first, the electronic device randomly generates two-dimensional Perlin noise with the same size as the first image, and performs binarization processing on the noise by using the first threshold to obtain an abnormal mask of the third image; a control factor is determined in a random manner, and a second image is obtained by formula (1).

[0084] In actual application, after the second image set is generated, the electronic device can construct a first loss function for direction constraint based on a regularization (DGR, Direction Guided Regularization) technique.

[0085] Specifically, in an embodiment, the generating the first loss function by using the first image set and the second image set comprises:

[0086] obtaining a second model, the second model being capable of guiding the first model to perform model training;

[0087] determine a first feature vector associated with normal regions of the first image set and a second feature vector associated with abnormal regions of the second image set in the second model;

[0088] determine a third feature vector associated with abnormal regions of the second image set in the first model;

[0089] generate the first loss function using the first feature vector, the second feature vector, and the third feature vector.

[0090] In actual application, the first model can be understood as a student model (denoted as S) associated with a knowledge distillation algorithm, and the second model can be understood as a trained teacher model (denoted as T) associated with the knowledge distillation algorithm, so that the first model can learn relevant feature distribution from the second model when the first model is trained, thereby realizing the guidance of the second model to the first model. In addition, the network structure adopted by the first model is the same as the network structure adopted by the second model, such as ResNet34 structure.

[0091] In actual application, the electronic device can train the second model using a relevant image data set (which can specifically include normal images and abnormal images) to obtain a trained second model; then, determine a first feature vector (also referred to as a normal feature vector) associated with normal regions and a second feature vector (also referred to as an abnormal feature vector) associated with abnormal regions in different feature layers of the trained second model, and determine a third feature vector associated with abnormal regions in different feature layers of the first model, the first feature vector representing the feature distribution of normal regions in the trained second model, the second feature vector representing the feature distribution of abnormal regions in the trained second model, and the third feature vector representing the feature distribution of abnormal regions in the first model; wherein the second feature vector and the third feature vector can be referred to as a synthetic abnormal feature pair or an abnormal feature vector pair, which can be mathematically represented as (f a s ,f a t ), which is not limited in the embodiments of the present application.

[0092] Here, after obtaining the first feature vector, the second feature vector, and the third feature vector, the electronic device can calculate the projection of the second feature vector and the third feature vector on the first feature vector, respectively, to construct the first loss function.

[0093] Specifically, in an embodiment, the generating the first loss function using the first feature vector, the second feature vector, and the third feature vector comprises:

[0094] a first value is obtained by using the first feature vector and the second feature vector, the first value representing a projection of the second feature vector on the first feature vector;

[0095] a second value is obtained by using the first feature vector and the third feature vector, the second value representing a projection of the third feature vector on the first feature vector;

[0096] the first loss function is generated by using the first value and the second value.

[0097] In actual application, since the first model and the second model have the same network structure, the number of feature layers representing the first model and the second model is the same; in this case, the electronic device can calculate the projection of the second feature vector corresponding to each feature layer on the first feature vector to obtain the first value corresponding to each feature layer, and at the same time, calculate the projection of the third feature vector corresponding to each feature layer on the first feature vector to obtain the second value corresponding to each feature layer; the first loss function is generated by using the first value and the second value corresponding to each feature layer.

[0098] Here, in the process of generating the first loss function, the electronic device can further combine the feature distribution difference between the normal area and the abnormal area in the second model to better constrain the training direction of the first model.

[0099] Specifically, in an embodiment, the first loss function is generated by using the first value and the second value, including:

[0100] a third value is obtained by using the second feature vector and the third feature vector, the third value representing the difference between the second feature vector and the third feature vector;

[0101] the first loss function is generated by using the first value, the second value and the third value.

[0102] In actual application, the third value can be obtained by calculating the cosine angle between the second feature vector and the third feature vector, and then the first loss function is constructed by combining the first value and the second value; wherein the first loss function can be represented as:

[0103]

[0104] wherein L represents the number of layers of the feature layers of the first model and the second model; cos() represents the cosine similarity function; represents the second feature vector corresponding to the lth feature layer of the second model, This is represented as the third feature vector corresponding to the l-th feature layer of the first model. f n t,l This is represented as the first feature vector corresponding to the l-th feature layer of the second model; Represented as the first value, Represented as the second value, Represented as The positive and negative signs,

[0105] In practical applications, for the second loss function, the electronic device can determine the second feature vector and the first feature vector corresponding to each feature layer of the trained second model, thereby constructing the second loss function; wherein, the second loss function can be expressed as:

[0106]

[0107] in, Let the fourth feature vector be the association between the l-th feature layer of the first model and the normal region. This fourth feature vector characterizes the feature distribution of the normal region in the first model. The fourth feature vector and the first feature vector can be called a normal feature vector pair, which can be mathematically represented as follows:

[0108] In practical applications, in step 103, the electronic device can use the first loss function and the second loss function to jointly constrain the training direction (or learning direction or optimization direction) of the first model; wherein, during model training, the first loss function at least increases the distance between the second feature vector and the third feature vector (or enhances the difference between the second feature vector and the third feature vector), and at the same time, makes the projection direction represented by the first value consistent with the projection direction represented by the second value; the second loss function at least decreases the distance between the first feature vector and the fourth feature vector (or reduces the difference between the first feature vector and the fourth feature vector).

[0109] Here, in order to increase the distance between the second feature vector and the third feature vector and to ensure the stability of the first model, the electronic device can adjust the distance between the second feature vector and the third feature vector based on the sign of the first value.

[0110] In an embodiment, when the first value is positive, a mirror direction of the second feature vector relative to the first feature vector is taken as an adjustment direction, and the distance between the third feature vector and the second feature vector is increased.

[0111] The mirror direction can be understood as a symmetric direction of the second feature vector relative to a symmetric axis in the direction of the first feature vector.

[0112] In actual application, when the first value is positive, the cosine angle between the first feature vector and the second feature vector is positive. In this case, the third feature vector is adjusted away from the second feature vector in the mirror direction by the first loss function. The distance between the third feature vector and the second feature vector can be increased by increasing the cosine angle between the third feature vector and the second feature vector.

[0113] In an embodiment, when the first value is negative, an opposite direction of the second feature vector relative to the first feature vector is taken as an adjustment direction, and the distance between the third feature vector and the second feature vector is increased.

[0114] The opposite direction can be understood as a direction opposite to the second feature vector.

[0115] In actual application, when the first value is negative, the cosine angle between the first feature vector and the second feature vector is negative. In this case, the third feature vector is adjusted away from the second feature vector in the opposite direction by the first loss function.

[0116] Based on the above model training method, an image detection method is further provided in the embodiments of the present application, which is applied to an electronic device, such as a server. Figure 2 As shown in the figure, the method comprises the following steps:

[0117] Step 201: obtaining a to-be-detected image;

[0118] Step 202: detecting the to-be-detected image by using a first model and a second model respectively to obtain a first detection result and a second detection result, the first model being obtained according to the model training method provided in the embodiments of the present application, the first detection result representing a detection result of the first model on the to-be-detected image, and the second detection result representing a detection result of the second model on the to-be-detected image;

[0119] Step 203: determining whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

[0120] In actual applications, the electronic device can further include a device capable of performing steps 201 to 203.

[0121] In actual applications, in step 202, the electronic device can obtain the first detection result and the second detection result by inputting the to-be-detected image (which can be represented as x) into the trained first model and the trained second model, respectively; the first detection result can include features corresponding to different feature layers of the first model, and the features of the lth feature layer of the first model can be represented as f t,l = T l (x); the second detection result can include features corresponding to different feature layers of the second model, and the features of the lth feature layer of the second model can be represented as f s,l = S l (x).

[0122] Here, the electronic device can calculate an anomaly value heat map corresponding to each feature layer using the first detection result and the second detection result; based on the anomaly value heat map of each feature layer, an anomaly value heat map of the to-be-detected image is obtained, and then it is determined whether the to-be-detected image is an abnormal image, and the anomaly value heat map can reflect the feature situation of the first model and the second model at different feature layers; wherein the anomaly value heat map of the to-be-detected image can be calculated using the following formula:

[0123]

[0124] Exemplarily, the electronic device can calculate an anomaly value heat map corresponding to each feature layer, and adjust the size of each feature layer corresponding to the anomaly value heat map, so that the size of each feature layer corresponding to the adjusted anomaly value heat map is the same as that of the to-be-detected image, and then the adjusted anomaly value heat map corresponding to each feature layer is accumulated to obtain the anomaly value heat map of the to-be-detected image.

[0125] In actual applications, in order to determine whether the to-be-detected image is an abnormal image, the electronic device can select an anomaly value with the largest value from the anomaly value heat map of the to-be-detected image, and take the selected anomaly value as the anomaly value (which can be represented as S AD ) of the to-be-detected image; in the case where the anomaly value of the to-be-detected image is greater than a preset second threshold value (which can be set as needed, and the embodiments of the present application do not limit it), the electronic device can determine that the to-be-detected image is an abnormal image; in the case where the anomaly value of the to-be-detected image is less than or equal to the second threshold value, the electronic device can determine that the to-be-detected image is a normal image.

[0126] In addition, in a case where the to-be-detected image is an abnormal image, the electronic device can further determine an abnormal region in the to-be-detected image based on the abnormal value heat map of the to-be-detected image. Specifically, the abnormal value heat map of the to-be-detected image is binarized by using a preset third threshold value (which can be set as needed, and the embodiments of the present application do not limit this), and the electronic device can obtain an abnormal segmentation mask (which can be denoted as S AL ) of the to-be-detected image, and further determine an abnormal region in the to-be-detected image. The abnormal segmentation mask is used to indicate whether a pixel in the to-be-detected image is a normal pixel or an abnormal pixel.

[0127] The model training and image detection method provided in the embodiments of the present application, wherein the model training method comprises: generating a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, and the second image set comprising one or more second images, each second image comprising one or more abnormal regions; generating a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region; training the first model by using the first loss function and a second loss function based on a knowledge distillation algorithm, the second loss function being capable of at least constraining a model training direction associated with a normal region, and the first model being used for abnormal detection of a to-be-detected image. In the process of training the student model by using the knowledge distillation algorithm, the optimization direction of the model training is determined based on the synthesized abnormal image and the normal image, and the student model is trained based on the determined optimization direction. In this way, the response difference of the student model and the teacher model in the knowledge distillation algorithm to the abnormal image can be strengthened, so that the student model and the teacher model after training can more accurately detect the abnormal region, thereby improving the accuracy of image abnormal detection.

[0128] The present application will be further described in detail below in combination with application examples.

[0129] In the application examples, an unsupervised image abnormal detection scheme based on direction-constrained knowledge distillation is proposed. Specifically, a student model is trained by synthesizing abnormal defect images and using a direction-constrained distillation method, so that the normal feature vector pairs of the student model and the teacher model are closely aligned, and the difference between the abnormal feature vector pairs is increased. Then, the trained student model and the teacher model are used to perform abnormal detection on images.

[0130] Specifically, as Figure 3As shown, the process of anomaly detection on images (which can also be referred to as pictures) contains two stages, namely a training stage and an inference stage; wherein the process of training the model in the training stage includes the following steps:

[0131] Step 1: Pre-training the teacher model T (i.e. the second model described above) using the relevant image classification dataset ImageNet;

[0132] Step 2: Knowledge distillation of the student model (i.e. the first model described above) using normal images in the anomaly detection image dataset (i.e. the first images described above) and the pre-trained teacher model; wherein the student model and the teacher model use the same network structure; the loss function (i.e. the second loss function described above) corresponding to knowledge distillation is formula (3).

[0133] Step 3: Synthesizing abnormal images (i.e. the second images described above) using normal images in the anomaly detection image dataset and images randomly selected from the texture image dataset (i.e. the third images described above);

[0134] Step 4: Constructing a direction-constrained regularization loss function (i.e. the first loss function described above) using the synthesized abnormal images and the corresponding normal images;

[0135] Here, in order to construct the regularization loss function, first, the projection (i.e. the first value described above) of the feature vector (i.e. the second feature vector described above) of the abnormal feature region of the teacher model in the feature vector (i.e. the first feature vector described above) of the normal feature region is calculated, and the positive and negative of the projection is calculated; second, the projection (i.e. the second value described above) of the feature vector (i.e. the third feature vector described above) of the abnormal feature region of the student model in the feature vector of the abnormal feature region of the teacher model is calculated; finally, based on the two projections calculated, the regularization loss function is constructed.

[0136] Step 5: Joint optimization of the loss function in step 2 and the regularization loss function in step 4 to train the student model S.

[0137] In the inference stage, the process of anomaly detection on the image to be detected includes the following steps:

[0138] Step 1: Inputting the image to be detected into the teacher model and the student model respectively to obtain features of different feature layers;

[0139] Step 2: Using the features of different feature layers of the teacher model and the student model to calculate the anomaly value heat map of the image to be detected;

[0140] Step 3: Setting the anomaly value heat map to the same size as the image to be detected;

[0141] Step 4: taking the maximum value in the anomaly value heat map as the anomaly value of the whole to-be-detected image; when the anomaly value of the whole to-be-detected image is greater than a threshold value, determining that the to-be-detected image is abnormal; otherwise, determining that the to-be-detected image is normal.

[0142] Step 5: setting a threshold value τ, performing threshold binarization on the anomaly value heat map of the to-be-detected image to obtain an anomaly segmentation mask of the image, so as to determine whether the pixels of the to-be-detected image are abnormal or normal.

[0143] In the application example, the projection of the abnormal feature of the teacher model and the student model on the normal feature is constructed through the synthesized abnormal image and the normal image, and the model optimization direction is dynamically adjusted according to the difference between the abnormal feature and the normal feature, so that the response difference of the teacher model and the student model to the abnormal area can be strengthened, the model can detect subtle and significant abnormalities (which can also be understood as defects), the problem of abnormal generalization is solved, and the stability of model optimization is improved.

[0144] In order to implement the model training method of the embodiment of the present application, the embodiment of the present application further provides a model training device arranged on an electronic device, as shown in the figure, the device comprises: Figure 4

[0145] The first generation unit 401 is configured to generate a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal areas, and the second image set comprising one or more second images, each second image comprising one or more abnormal areas.

[0146] The second generation unit 402 is configured to generate a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal area.

[0147] The training unit 403 is configured to train the first model based on a knowledge distillation algorithm by using the first loss function and a second loss function, the second loss function being capable of at least constraining a model training direction associated with a normal area, and the first model being used for abnormal detection of a to-be-detected image.

[0148] In an embodiment, the second generation unit 402 is configured to:

[0149] obtain a second model, the second model being capable of at least guiding the first model to perform model training;

[0150] determine a first feature vector associated with a normal area of the first image set and a second feature vector associated with an abnormal area of the second image set in the second model; ​

[0151] determining a third feature vector associated with the abnormal region of the second image set in the first model;

[0152] generating the first loss function by using the first feature vector, the second feature vector and the third feature vector.

[0153] In an embodiment, the second generating unit 402 is configured to:

[0154] obtaining a first value representing a projection of the second feature vector on the first feature vector by using the first feature vector and the second feature vector;

[0155] obtaining a second value representing a projection of the third feature vector on the first feature vector by using the first feature vector and the third feature vector;

[0156] generating the first loss function by using the first value and the second value.

[0157] In an embodiment, the second generating unit 402 is configured to:

[0158] obtaining a third value representing a difference degree between the second feature vector and the third feature vector by using the second feature vector and the third feature vector;

[0159] generating the first loss function by using the first value, the second value and the third value.

[0160] In an embodiment, the training unit 403 is configured to increase a distance between the second feature vector and the third feature vector by using the first loss function in a model training process.

[0161] In an embodiment, the training unit 403 is configured to, in a case that the first value is a positive number, increase the distance between the third feature vector and the second feature vector by taking a mirror direction of the second feature vector relative to the first feature vector as an adjustment direction.

[0162] In an embodiment, the training unit 403 is configured to, in a case that the first value is a negative number, increase the distance between the third feature vector and the second feature vector by taking an opposite direction of the second feature vector relative to the first feature vector as an adjustment direction.

[0163] In an embodiment, the first generating unit 401 is configured to:

[0164] obtaining a third image set, the third image set containing one or more third images, each third image containing an abnormal region;

[0165] selecting one first image from the first image set and one or more third images from the third image set;

[0166] performing fusion processing on the selected one or more third images and the first image to obtain a second image.

[0167] In an embodiment, the first generating unit 401 is configured to:

[0168] determine a target position of each of the selected one or more third images relative to the first image;

[0169] perform fusion processing on the selected one or more third images and the first image based on the determined target position to obtain a second image.

[0170] In actual application, the first generating unit 401 and the second generating unit 402 can be implemented by a communication interface combined with a processor in a model training device, and the training unit 403 can be implemented by a processor in the model training device.

[0171] It should be noted that the model training device provided in the above embodiments is only used as an example for the division of the above program units during model training. In actual application, the above processing can be completed by different program units according to needs, that is, the internal structure of the device is divided into different program units to complete all or part of the above processing. In addition, the model training device and the model training method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0172] In order to implement the image detection method of the embodiments of the present application, the embodiments of the present application further provide an image detection device arranged on an electronic device, as shown in the following Figure 5 The device includes:

[0173] The obtaining unit 501 is configured to obtain a to-be-detected image.

[0174] The detection unit 502 is configured to detect the to-be-detected image by using a first model and a second model to obtain a first detection result and a second detection result, the first model being obtained based on the model training method provided in the embodiments of the present application, the first detection result representing a detection result of the first model on the to-be-detected image, and the second detection result representing a detection result of the second model on the to-be-detected image.

[0175] The determination unit 503 is configured to determine whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

[0176] It should be noted that the image detection device provided in the above embodiment is only exemplified by the division of the above program units when performing image detection. In actual application, the above processing can be completed by different program units according to needs, that is, the internal structure of the device is divided into different program units to complete all or part of the above-described processing. In addition, the image detection device and the image detection method provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0177] Based on the hardware implementation of the above program modules, and in order to realize the method of model training and image detection of the embodiment of the application, the embodiment of the application further provides an electronic device, as shown in the figure. Figure 6 As shown in the figure, the electronic device 600 includes:

[0178] The communication interface 601 can interact with other devices;

[0179] The processor 602 is connected with the communication interface 601 to realize interaction with other devices, and is used to run a computer program to execute the method provided in one or more technical solutions described above;

[0180] The memory 603 stores the computer program.

[0181] Specifically, in the process of model training provided in the embodiment of the application, the processor 602 is configured to generate a second image set by using a first image set, the first image set contains one or more first images, each first image contains one or more normal regions, the second image set contains one or more second images, each second image contains one or more abnormal regions; generate a first loss function by using the first image set and the second image set, the first loss function can at least constrain the model training direction associated with the abnormal region; and train the first model based on a knowledge distillation algorithm, using the first loss function and a second loss function, the second loss function can at least constrain the model training direction associated with the normal region, and the first model is used for abnormal detection of the to-be-detected image;

[0182] In an embodiment, the processor 602 is configured to:

[0183] Obtain a second model through the communication interface 601, the second model can at least guide the first model to perform model training; determine a first feature vector associated with the normal region of the first image set and a second feature vector associated with the abnormal region of the second image set in the second model;

[0184] determining a third feature vector associated with the abnormal region of the second image set in the first model;

[0185] generating the first loss function by using the first feature vector, the second feature vector and the third feature vector.

[0186] In an embodiment, the processor 602 is configured to:

[0187] obtaining a first value by using the first feature vector and the second feature vector, the first value representing a projection of the second feature vector on the first feature vector;

[0188] obtaining a second value by using the first feature vector and the third feature vector, the second value representing a projection of the third feature vector on the first feature vector;

[0189] generating the first loss function by using the first value and the second value.

[0190] In an embodiment, the processor 602 is configured to:

[0191] obtaining a third value by using the second feature vector and the third feature vector, the third value representing a difference degree between the second feature vector and the third feature vector;

[0192] generating the first loss function by using the first value, the second value and the third value.

[0193] In an embodiment, the processor 602 is configured to, in a model training process, increase a distance between the second feature vector and the third feature vector by using the first loss function.

[0194] In an embodiment, the processor 602 is configured to, in a case that the first value is positive, increase the distance between the third feature vector and the second feature vector by taking a mirror direction of the second feature vector relative to the first feature vector as an adjustment direction.

[0195] In an embodiment, the processor 602 is configured to, in a case that the first value is negative, increase the distance between the third feature vector and the second feature vector by taking an opposite direction of the second feature vector relative to the first feature vector as an adjustment direction.

[0196] In an embodiment, the processor 602 is configured to:

[0197] obtaining a third image set by using the communication interface 601, the third image set containing one or more third images, each third image containing an abnormal region;

[0198] Select a first image from the first image set, and select one or more third images from the third image set;

[0199] The selected third image and the first image are fused together to obtain a second image.

[0200] In one embodiment, the processor 602 is configured to:

[0201] Determine the target position of each of the selected one or more third images relative to the first image;

[0202] Using the determined target location, one or more selected third images and the first image are fused together to obtain a second image.

[0203] In the image detection process provided in this application embodiment, the processor 602 is used for:

[0204] The processor 602 is configured to acquire an image to be detected through a communication interface; detect the image to be detected using a first model and a second model respectively to obtain a first detection result and a second detection result, wherein the first model is obtained based on the model training method provided in the embodiments of this application, the first detection result represents the detection status of the first model on the image to be detected, and the second detection result represents the detection status of the second model on the image to be detected; and determine whether the image to be detected is an abnormal image using the first detection result and the second detection result.

[0205] It should be noted that the specific processing procedures of the communication interface 601 and the processor 602 can be understood with reference to the above method.

[0206] Of course, in practical applications, the various components in electronic device 600 are coupled together through bus system 604. It can be understood that bus system 604 is used to realize the connection and communication between these components. In addition to a data bus, bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0207] The memory 603 in this embodiment is used to store various types of data to support the operation of the electronic device 600. Examples of such data include any computer program used to operate on the electronic device 600.

[0208] The methods disclosed in the embodiments of this application can be applied to the processor 602, or implemented by the processor 602. The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 602 or by instructions in the form of software. The processor 602 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 603. The processor 602 reads the information in the memory 603 and combines its hardware to complete the steps of the aforementioned method.

[0209] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0210] It is understood that the memory (memory 603) in this embodiment of the application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0211] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 603 storing a computer program. This computer program can be executed by the processor 602 of the electronic device 600 to complete the steps of the aforementioned model training method or the steps of the aforementioned image detection method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0212] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 602 of an electronic device 600 to complete the steps of the aforementioned model training method or the steps of the aforementioned image detection method.

[0213] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0214] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0215] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method comprises: generating a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, the second image set comprising one or more second images, each second image comprising one or more abnormal regions; generating a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region; training the first model by using the first loss function and a second loss function based on a knowledge distillation algorithm, the second loss function being capable of at least constraining a model training direction associated with a normal region, the first model being used for abnormality detection of a to-be-detected image.

2. The method of claim 1, wherein, The method of generating the first loss function by using the first image set and the second image set comprises: obtaining a second model, the second model being capable of at least guiding the first model to perform model training; determining a first feature vector associated with a normal region of the first image set and a second feature vector associated with an abnormal region of the second image set in the second model; determining a third feature vector associated with an abnormal region of the second image set in the first model; generating the first loss function by using the first feature vector, the second feature vector, and the third feature vector.

3. The method of claim 2, wherein, The method of generating the first loss function by using the first feature vector, the second feature vector, and the third feature vector comprises: obtaining a first value by using the first feature vector and the second feature vector, the first value representing a projection of the second feature vector on the first feature vector; obtaining a second value by using the first feature vector and the third feature vector, the second value representing a projection of the third feature vector on the first feature vector; generating the first loss function by using the first value and the second value.

4. The method of claim 3, wherein, The method of generating the first loss function by using the first value and the second value comprises: obtaining a third value by using the second feature vector and the third feature vector, the third value representing a difference degree between the second feature vector and the third feature vector; generating the first loss function by using the first value, the second value, and the third value.

5. The method of claim 3, wherein, in the model training process, the distance between the second feature vector and the third feature vector is increased by using the first loss function.

6. The method of claim 5, wherein, in a case where the first value is positive, a mirror direction of the second feature vector relative to the first feature vector is taken as an adjustment direction, and the distance between the third feature vector and the second feature vector is increased.

7. The method of claim 5, wherein, in a case where the first value is negative, an opposite direction of the second feature vector relative to the first feature vector is taken as an adjustment direction, and the distance between the third feature vector and the second feature vector is increased.

8. The method of claim 1, wherein, The generating the second image set by using the first image set comprises: obtaining a third image set, the third image set comprising one or more third images, each third image comprising an abnormal region; selecting a first image from the first image set and selecting one or more third images from the third image set; performing fusion processing on the selected one or more third images and the first image to obtain a second image.

9. The method of claim 8, wherein, The performing fusion processing on the selected one or more third images and the first image to obtain a second image comprises: determining a target position of each of the selected one or more third images relative to the first image; performing fusion processing on the selected one or more third images and the first image by using the determined target position to obtain a second image.

10. An image detection method characterized by, comprising: obtaining a to-be-detected image; detecting the to-be-detected image by using a first model and a second model to obtain a first detection result and a second detection result, the first model being obtained according to the method in any one of claims 1 to 9, the first detection result representing a detection situation of the first model on the to-be-detected image, and the second detection result representing a detection situation of the second model on the to-be-detected image; determining whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

11. A model training apparatus, comprising: comprising: a first generating unit configured to generate a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, and the second image set comprising one or more second images, each second image comprising one or more abnormal regions; a second generating unit configured to generate a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region; a training unit configured to train a first model by using the first loss function and a second loss function based on a knowledge distillation algorithm, the second loss function being capable of at least constraining a model training direction associated with a normal region, and the first model being used for abnormal detection on a to-be-detected image.

12. An image detection apparatus characterized by comprising: comprising: an obtaining unit configured to obtain a to-be-detected image; a detection unit configured to detect the to-be-detected image by using a first model and a second model to obtain a first detection result and a second detection result, the first model being obtained according to the method in any one of claims 1 to 9, the first detection result representing a detection situation of the first model on the to-be-detected image, and the second detection result representing a detection situation of the second model on the to-be-detected image; a determination unit configured to determine whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

13. An electronic device, comprising: comprising: a processor and a communication interface; wherein The processor is configured to generate a second image set by using a first image set, the first image set comprising one or more first images, each first image comprising one or more normal regions, the second image set comprising one or more second images, each second image comprising one or more abnormal regions; generate a first loss function by using the first image set and the second image set, the first loss function being capable of at least constraining a model training direction associated with an abnormal region; and train the first model based on a knowledge distillation algorithm, by using the first loss function and a second loss function, the second loss function being capable of at least constraining a model training direction associated with a normal region, the first model being used for performing abnormal detection on a to-be-detected image. Alternatively, The processor is configured to obtain a to-be-detected image through a communication interface; perform detection on the to-be-detected image by using a first model and a second model respectively, to obtain a first detection result and a second detection result, the first model being obtained according to the method of any one of claims 1 to 9, the first detection result representing a detection result of the first model on the to-be-detected image, and the second detection result representing a detection result of the second model on the to-be-detected image; and determine whether the to-be-detected image is an abnormal image by using the first detection result and the second detection result.

14. An electronic device, comprising: Comprise: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute the steps of the method of any one of claims 1 to 9, or the steps of the method of claim 10 when running the computer program.

15. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 9, or the steps of the method of claim 10.

16. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 9, or the steps of the method of claim 10. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 9, or the steps of the method of claim 10.

Images