Abnormality detection method, device, electronic device and computer readable storage medium

Abstract

This disclosure provides an anomaly detection method, apparatus, electronic device, and computer-readable storage medium. The method includes: training an initial detection network using a first sample set to obtain a detection network; the first sample set being a positive sample set; during anomaly detection of multiple images to be detected by the detection network, obtaining a second sample set based on the detected normal image set and abnormal image set, the second sample set being an incremental sample set containing positive and negative samples; wherein the negative samples are abnormal images containing anomalies; and updating and training the detection network using the second sample set to obtain an updated detection network. This disclosure improves the accuracy and flexibility of anomaly detection.

Description

Technical Field

[0001] The present disclosure relates to machine vision technology, and in particular to an abnormality detection method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In recent years, deep learning algorithms have made great progress in various fields and have also been implemented in many industrial visual inspection fields. Related technologies use deep learning algorithm training to obtain anomaly detection models for abnormal image detection. Deep learning algorithms often require a large number of high-quality training samples. However, the model structure and learning strategy applicable to normal sample learning methods and abnormal sample learning methods are different. Therefore, the current deep learning methods have poor compatibility with different types of training samples, resulting in a relatively single training sample that can be used. This not only reduces the flexibility of anomaly detection, but also reduces the accuracy of anomaly detection. Summary of the invention

[0003] The embodiments of the present disclosure provide an anomaly detection method, an apparatus, an electronic device, and a computer-readable storage medium, which can improve the accuracy and flexibility of anomaly detection.

[0004] The technical solution of the embodiment of the present disclosure is implemented as follows:

[0005] The embodiment of the present disclosure provides an anomaly detection method, including: using a first sample set to train an initial detection network to obtain a detection network; the first sample set is a positive sample set; in the process of the detection network performing anomaly detection on multiple images to be detected, based on the detected normal image set and abnormal image set, a second sample set is obtained, and the second sample set is an incremental sample set including positive samples and negative samples; wherein the negative sample is an abnormal image with an abnormality in the image; using the second sample set, the detection network is updated and trained to obtain an updated detection network.

[0006] In the above method, the first positive sample set is used to train the initial detection network to obtain the detection network, including: using the initial detection network to detect the positive samples in the first sample set to obtain the first feature difference sequence corresponding to the positive sample; determining the training loss corresponding to the positive sample based on the first feature difference sequence and the normal loss function; based on the training loss, the initial detection network is trained and adjusted until the final loss obtained is less than a preset loss threshold, thereby obtaining the detection network; wherein the normal loss function characterizes that the reconstructed feature sequence corresponding to the positive sample is closer to the positive sample.

[0007] In the above method, obtaining the second sample set based on the detected normal image set and abnormal image set includes: determining the image to be verified from the detected normal image set and abnormal image set; and verifying the image to be verified to obtain the second sample set.

[0008] In the above method, the images to be verified include: at least one abnormal image and at least one normal image; the verifying of the images to be verified to obtain the second sample set includes: verifying the at least one abnormal image and the at least one normal image respectively to obtain respective verification results; taking the abnormal images correctly represented by the verification results as the negative samples, and taking the normal images correctly represented by the verification results as the positive samples; correctly marking the abnormal images with incorrect verification results and the normal images with incorrect verification results to obtain the positive samples and the negative samples; and determining the set of the negative samples and the positive samples as the second sample set.

[0009] In the above method, after the initial detection network is trained using the first sample set to obtain the detection network, the method further includes: using the detection network to perform anomaly detection on each image to be detected to obtain an anomaly score at each pixel position of each image to be detected; based on the anomaly score, drawing an anomaly detection image corresponding to each image to be detected; and obtaining the normal image set and the abnormal image set in multiple images to be detected based on the abnormal detection image.

[0010] In the above method, the normal image set and the abnormal image set in multiple images to be detected are obtained based on the abnormal detection image, including: when the abnormal detection image corresponding to each image to be detected is represented and each image to be detected is an abnormal image, each image to be detected is used as an abnormal image, and after traversing the multiple images to be detected, the abnormal image set containing at least one abnormal image is obtained; when the abnormal detection image corresponding to each image to be detected is represented and each image to be detected is a normal image, each image to be detected is used as a normal image, and after traversing the multiple images to be detected, the normal image set containing at least one normal image is obtained.

[0011] In the above method, determining the image to be verified from the detected normal image set and the abnormal image set includes: determining a first maximum abnormality score that is the largest among the abnormality scores at each pixel position from each abnormal image in the abnormal image set; determining a second maximum abnormality score that is the largest among the abnormality scores at each pixel position from each normal image in the normal image set; when the first maximum abnormality score belongs to a first preset numerical range of a preset abnormality threshold, determining that the abnormal image corresponding to the first maximum abnormality score belongs to the image to be verified; when the second maximum abnormality score belongs to a second preset numerical range of the preset abnormality threshold, determining that the normal image corresponding to the second maximum abnormality score belongs to the image to be verified.

[0012] In the above method, determining the images to be verified from the detected normal image set and the abnormal image set includes: determining all images in the normal image set and the abnormal image set as the images to be verified.

[0013] In the above method, the use of the second sample set to update the detection network to obtain an updated detection network includes: when the positive samples in the second sample set are used to train the detection network, the detection network is used to obtain a first feature difference sequence corresponding to the positive sample, and the first training loss corresponding to the positive sample is determined according to the first feature difference sequence and the normal loss function; when the negative samples in the second sample set are used to train the detection network, the detection network is used to obtain a second feature difference sequence corresponding to the negative sample, and the second training loss corresponding to the negative sample is determined according to the second feature difference sequence, the true value of the preset negative sample and the abnormal loss function; based on the first training loss and the second training loss, the detection network is trained and adjusted respectively until the final loss obtained is less than the preset loss threshold, and the updated detection network is obtained; wherein the normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is close to the positive sample; the abnormal loss function represents that the reconstructed feature sequence corresponding to the abnormal part in the negative sample is pushed away from the abnormal part, and the reconstructed feature sequence corresponding to the normal part in the negative sample is close to the normal part.

[0014] In the above method, the updated detection network includes: an updated feature extraction network and an updated reconstruction network; after the detection network is updated and trained using the second sample set to obtain the updated detection network, the method further includes: using the updated feature extraction network to perform feature processing of different scales on the target image to obtain a multi-scale feature sequence; using the updated reconstruction network to reconstruct the multi-scale feature sequence and a preset query word sequence to obtain a reconstructed feature sequence; determining the feature difference at each pixel position of the target image based on the reconstructed feature sequence and the multi-scale feature sequence; determining the anomaly score at each pixel position based on the feature difference, and drawing the anomaly detection image corresponding to the target image based on the anomaly score.

[0015] The disclosed embodiment provides an anomaly detection device, comprising: a first training unit, used to train an initial detection network with a first sample set to obtain a detection network; the first sample set is a positive sample set; an acquisition unit, used to obtain a second sample set based on a detected normal image set and an abnormal image set during anomaly detection of a plurality of images to be detected by the detection network, wherein the second sample set is an incremental sample set including positive samples and negative samples; wherein the negative sample is an abnormal image with an abnormality in the image; and a second training unit, used to update the detection network with the second sample set to obtain an updated detection network.

[0016] In the above-mentioned device, the first training unit is also used to use the initial detection network to detect the positive samples in the first sample set to obtain the first feature difference sequence corresponding to the positive sample; determine the training loss corresponding to the positive sample according to the first feature difference sequence and the normal loss function; based on the training loss, the initial detection network is trained and adjusted until the final loss obtained is less than the preset loss threshold, and the detection network is obtained; wherein the normal loss function characterizes that the reconstructed feature sequence corresponding to the positive sample is closer to the positive sample.

[0017] In the above device, the acquisition unit is further used to determine the image to be verified from the detected normal image set and the abnormal image set; and verify the image to be verified to obtain the second sample set.

[0018] In the above-mentioned device, the images to be verified include: at least one abnormal image and at least one normal image; the acquisition unit is also used to verify the at least one abnormal image and the at least one normal image respectively to obtain respective verification results; the abnormal image represented by the verification result correctly is used as the negative sample, and the normal image represented by the verification result correctly is used as the positive sample; the abnormal image represented by the verification result incorrectly and the normal image represented by the verification result incorrectly are correctly labeled to obtain the positive sample and the negative sample; the set of the negative sample and the positive sample is determined as the second sample set.

[0019] In the above-mentioned device, the anomaly detection device also includes: a detection unit; the detection unit is further used to use the first sample set to train the initial detection network, and after obtaining the detection network, use the detection network to perform anomaly detection on each image to be detected to obtain an anomaly score at each pixel position of each image to be detected; based on the anomaly score, draw an anomaly detection image corresponding to each image to be detected; the acquisition unit is also used to obtain the normal image set and the abnormal image set in multiple images to be detected based on the abnormal detection image.

[0020] In the above-mentioned device, the acquisition unit is also used to represent the abnormal detection image corresponding to each image to be detected, and when each image to be detected is an abnormal image, take each image to be detected as an abnormal image, and after traversing the multiple images to be detected, obtain the abnormal image set containing at least one abnormal image; when representing the abnormal detection image corresponding to each image to be detected, and when each image to be detected is a normal image, take each image to be detected as a normal image, and after traversing the multiple images to be detected, obtain the normal image set containing at least one normal image.

[0021] In the above device, the acquisition unit is further used to determine a first maximum abnormality score that is the largest among the abnormality scores at each pixel position from each abnormal image in the abnormal image set; determine a second maximum abnormality score that is the largest among the abnormality scores at each pixel position from each normal image in the normal image set; when the first maximum abnormality score belongs to a first preset numerical range of a preset abnormal threshold, determine that the abnormal image corresponding to the first maximum abnormality score belongs to the image to be verified; when the second maximum abnormality score belongs to a second preset numerical range of the preset abnormal threshold, determine that the normal image corresponding to the second maximum abnormality score belongs to the image to be verified.

[0022] In the above device, the acquisition unit is further used to determine all images in the normal image set and the abnormal image set as the images to be verified.

[0023] In the above-mentioned device, the second training unit is also used for, when the positive samples in the second sample set are used to train the detection network, obtaining the first feature difference sequence corresponding to the positive sample by using the detection network, and determining the first training loss corresponding to the positive sample according to the first feature difference sequence and the normal loss function; when the negative samples in the second sample set are used to train the detection network, obtaining the second feature difference sequence corresponding to the negative sample by using the detection network, and determining the second training loss corresponding to the negative sample according to the second feature difference sequence, the true value of the preset negative sample and the abnormal loss function; based on the first training loss and the second training loss, respectively training and adjusting the detection network until the final loss obtained is less than the preset loss threshold, the updated detection network is obtained; wherein the normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is close to the positive sample; the abnormal loss function represents that the reconstructed feature sequence corresponding to the abnormal part in the negative sample is pushed away from the abnormal part, and the reconstructed feature sequence corresponding to the normal part in the negative sample is close to the normal part.

[0024] In the above device, the updated detection network includes: an updated feature extraction network and an updated reconstruction network; the detection unit is further used to use the second sample set to update the detection network, and after obtaining the updated detection network, use the updated feature extraction network to perform feature processing of different scales on the target image to obtain a multi-scale feature sequence; use the updated reconstruction network to reconstruct the multi-scale feature sequence and a preset query word sequence to obtain a reconstructed feature sequence; determine the feature difference at each pixel position of the target image based on the reconstructed feature sequence and the multi-scale feature sequence; determine the anomaly score at each pixel position based on the feature difference, and draw the anomaly detection image corresponding to the target image based on the anomaly score.

[0025] An embodiment of the present disclosure provides an electronic device, including: a memory, used to store executable instructions; and a processor, used to implement the above-mentioned abnormality detection method when executing the executable instructions stored in the memory.

[0026] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for implementing the above-mentioned anomaly detection method when executed by a processor.

[0027] The embodiments of the present disclosure have the following beneficial effects:

[0028] The initial detection network is trained with a positive sample set to obtain a detection network. In the process of the detection network performing anomaly detection on multiple images to be detected, an incremental sample set including positive samples and negative samples is obtained based on the detected normal image set and abnormal image set, wherein the negative sample is an abnormal image with an abnormality in the image. The incremental sample set including positive samples and negative samples is used to update the detection network to obtain an updated detection network. Therefore, the obtained updated detection network is more suitable for actual detection scenarios, and the detection accuracy of the updated detection network is higher, so that when the updated detection network is subsequently used to perform anomaly detection on the images to be detected, the detection results obtained are more accurate. At the same time, since the incremental samples include positive samples and negative samples, the obtained updated detection network can be obtained by training with incremental samples of positive samples and negative samples, so that the detection network can be compatible with different situations such as positive samples and negative samples, thereby improving the versatility and flexibility of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 An optional flowchart of an abnormality detection method provided in an embodiment of the present disclosure;

[0030] Figure 2 A schematic diagram of a negative sample set exemplarily including three negative samples provided for an embodiment of the present disclosure;

[0031] Figure 3 An optional flowchart of an anomaly detection method provided in an embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of a process for obtaining a multi-scale feature sequence of a target image I provided in an embodiment of the present disclosure;

[0033] Figure 5 is a schematic diagram of a reconstructed network structure of an encoder-decoder structure provided by an embodiment of the present disclosure;

[0034] Figure 6 A schematic diagram of a process for exemplarily obtaining reconstruction features of an image provided in an embodiment of the present disclosure;

[0035] Figure 7 A schematic diagram of the effect of an abnormality detection image A' obtained after an abnormality detection is performed on a target image A, provided by an embodiment of the present disclosure;

[0036] Figure 8 An optional flowchart of an abnormality detection method provided in an embodiment of the present disclosure;

[0037] Fig. 9A schematic diagram of a process for obtaining a training loss corresponding to a positive sample by reconstructing a sample feature difference sequence between a sample feature sequence and a multi-scale sample feature sequence provided in an embodiment of the present disclosure;

[0038] Fig.10 An optional flowchart of an abnormality detection method provided in an embodiment of the present disclosure;

[0039] Fig.11 An optional flowchart of an abnormality detection method provided in an embodiment of the present disclosure;

[0040] Fig.12 An optional flowchart of an abnormality detection method provided in an embodiment of the present disclosure;

[0041] Fig.13 A schematic diagram of a flow chart of exemplarily obtaining an updated reconstructed network provided in an embodiment of the present disclosure;

[0042] Fig.14 A schematic diagram of the structure of an abnormality detection device provided in an embodiment of the present disclosure;

[0043] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present disclosure. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0048] The embodiments of the present disclosure provide an anomaly detection method that can improve the accuracy and flexibility of anomaly detection. The anomaly detection method provided by the embodiments of the present disclosure is applied to electronic devices. The electronic devices provided by the embodiments of the present disclosure can be implemented as various types of user terminals such as AR glasses, laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers.

[0049] Figure 1 is an optional flow chart of the anomaly detection method provided by the embodiment of the present disclosure, which is combined with Figure 1 The steps shown are explained.

[0050] S101. Use a first sample set to train an initial detection network to obtain a detection network; the first sample set is a positive sample set.

[0051] The anomaly detection method in the embodiments of the present disclosure is applicable to scenarios where anomalies are discovered through image detection. For example, by performing anomaly detection on product images on a production line, abnormal products such as defective products are discovered; or, by performing anomaly detection on batch images, abnormal images that do not meet preset overall specifications are discovered. The specific method can be selected according to actual conditions and is not limited by the embodiments of the present disclosure.

[0052] In the embodiment of the present disclosure, the positive sample set contains at least one positive sample. A positive sample is a sample in which every pixel in the image is normal; an image with abnormal parts in the image is called an abnormal image, and the opposite of a positive sample is a negative sample, which is actually an abnormal image. For example, Figure 2 A negative sample set including three negative samples is shown, wherein negative sample 2-1 is an image of a germinated seed, negative sample 2-2 is an image of a toothbrush with abnormal bristles at the tip of the brush head, and negative sample 2-3 is an image of a broken nut. Figure 2 2-10, 2-20 and 2-30 respectively show the areas containing abnormalities (ie, abnormal pixels) in negative sample 2-1, negative sample 2-2 and negative sample 2-3.

[0053] In the disclosed embodiment, the electronic device may use a positive sample set including at least one positive sample to continuously train the initial detection network until a trained detection network is obtained.

[0054] In some embodiments, the initial detection network may be composed of an initial feature extraction network and an initial reconstruction network, and the electronic device may continuously train the initial feature extraction network and the initial reconstruction network simultaneously or separately to obtain a trained feature extraction network and a trained reconstruction network, and use the obtained trained feature extraction network and the trained reconstruction network as the obtained detection network. In other embodiments, the initial detection network may also be composed of a pre-trained feature extraction network and an initial reconstruction network, and the electronic device may continuously train only the initial reconstruction network to obtain a trained reconstruction network, and use the pre-trained feature extraction network and the trained reconstruction network as the obtained detection network.

[0055] In some embodiments, the electronic device may use a deep convolutional neural network (CNN) network including multiple feature extraction layers, such as ResNet-34, as a feature extraction network.

[0056] In some embodiments, the reconstruction network can be implemented by a neural network with an encoder-decoder (Transformer) structure. Among them, Transformer is a deep neural network with a global attention mechanism, which first appeared in the field of natural language processing. In recent years, Transformer has gradually been widely used in computer vision tasks due to its powerful representation ability. At present, the application of Transformer has not been involved in anomaly detection scenarios, and the disclosed embodiments can extend the application of Transformer to anomaly detection scenarios, using Transformer to capture the global attention of the input sequence, and strengthen the reconstruction network's deep understanding of semantic information, so that the reconstruction network can realize feature reconstruction on the premise of understanding the deep semantic features of the image.

[0057] S102. In the process of performing anomaly detection on multiple images to be detected by the detection network, a second sample set is obtained based on the detected normal image set and abnormal image set, and the second sample set is an incremental sample set including positive samples and negative samples; wherein the negative sample is an abnormal image with an abnormality in the image.

[0058] In the embodiments of the present disclosure, the image to be detected can be any image, for example, it can be the product image on the above-mentioned production line, and the embodiments of the present disclosure are not limited to this. In some embodiments, the electronic device can obtain the image of the object to be detected as the image to be detected through an image acquisition device, for example, by performing image acquisition on the products produced on the production line to obtain the image to be detected, so as to perform abnormality detection on the image to be detected by the method of the embodiments of the present disclosure, so as to determine whether there is an abnormality in the object to be detected; or, the electronic device can also directly obtain the image to be detected from other devices. The electronic device can obtain one image to be detected at a time, or multiple images to be detected at a time, and the embodiments of the present disclosure are not limited to this.

[0059] In the disclosed embodiments, the electronic device may use a detection model trained by a positive sample set to perform anomaly detection on multiple images to be detected, obtain a normal image set and an abnormal image set, and obtain an incremental sample set consisting of the normal image set and the abnormal image set based on the obtained normal image set and abnormal image set, wherein the normal image set contains at least one normal image and the abnormal image set contains at least one abnormal image.

[0060] In some embodiments, the electronic device may use all normal images detected as a normal image set, and use all abnormal images detected as an abnormal image set. In some embodiments, the electronic device may also select some normal images and abnormal images from all normal images and all abnormal images detected, thereby obtaining a normal image set and an abnormal image set, which is not limited in the embodiments of the present disclosure.

[0061] S103: Use the second sample set to update the detection network and obtain an updated detection network.

[0062] In the disclosed embodiment, the electronic device may use an incremental sample set including positive samples and negative samples to continue updating and training the obtained detection network until the final loss reaches a preset loss threshold, thereby obtaining a trained detection network, and using the trained detection network as the updated detection network.

[0063] In some embodiments, after the above S103, S104-S107 may also be performed; Figure 3 is an optional flow chart of the anomaly detection method provided by the embodiment of the present disclosure, which is combined with Figure 3 The steps shown are explained.

[0064] S104, using the updated feature extraction network to perform feature processing of different scales on the target image to obtain a multi-scale feature sequence; the updated detection network includes: an updated feature extraction network and an updated reconstruction network.

[0065] In the embodiment of the present disclosure, the target image may be any image, for example, it may be the product image on the above-mentioned production line, and the embodiment of the present disclosure is not limited to this.

[0066] In the disclosed embodiment, when the updated detection network includes: an updated feature extraction network and an updated reconstruction network, the electronic device can use the updated feature extraction network to extract features of different scales on the target image, and obtain features of multiple scales of the target image at multiple scales; and fuse the features of multiple scales to obtain multi-scale features; perform shape transformation on the multi-scale features, convert the multi-dimensional feature information into a sequence form, and obtain a multi-scale feature sequence. For example, when ResNet-34 is used as the feature extraction network, the electronic device can use the updated ResNet-34 to extract features of multiple scales on the target image, use the bottom feature extraction layer of ResNet-34 to obtain the apparent features of the target image, such as color, boundary division, contrast, brightness, etc., and use the high-level extraction layer of ResNet-34 to output the high-level semantic information of the target image; the electronic device uses the apparent features of the target image and the high-level semantic information as features of multiple scales, and fuses the features to obtain multi-scale features.

[0067] For example, Figure 4 As shown, the electronic device uses the updated CNN network as a feature extraction network to extract features of the target image I at four scales, and obtains features B(I) at four scales, where: B(I)=f1,f2,f3,f4. Here, i=1,2,3,4, i represents the i-th network layer of the CNN network. Figure 4 It can be seen that the feature sizes of scale features extracted by different network layers are different. The electronic device can first adjust the features of different sizes to the same size, such as adjusting to the size H×W, and then perform feature fusion on the four features after the adjustment to obtain multi-scale features, as shown in formula (1):

[0068]

[0069] In formula (1), R represents the resizing operation, and C = ∑ i C i The feature channel representing the multi-scale feature f. It can be seen that in the disclosed embodiment, the features of multiple scales obtained by different feature extraction layers have different levels of receptive fields and are very sensitive to abnormal areas of different degrees that may exist in the target image. Therefore, anomaly detection based on multi-scale features is more helpful to improve the accuracy of anomaly detection.

[0070] In the disclosed embodiment, the electronic device can perform shape transformation on the multi-scale features, thereby converting the multi-scale features into a sequence form as an input for a subsequent reconstruction network. Figure 4 As shown, for a multi-scale feature f with a dimension of H×W×C, the electronic device can merge the H×W dimensions corresponding to each channel to achieve a shape transformation of the multi-scale feature, and obtain H×W C-dimensional multi-scale feature vectors as a multi-scale feature sequence.

[0071] In some embodiments, the multi-scale feature sequence can be a feature word sequence containing any form of word vectors, such as a word2vec sequence, a GloVe sequence, a one-hot encoding sequence, etc. The specific selection is based on actual conditions and is not limited in the embodiments of the present disclosure.

[0072] S105 , using the updated reconstruction network to reconstruct the multi-scale feature sequence and the preset query word sequence to obtain a reconstructed feature sequence.

[0073] In the disclosed embodiment, the electronic device can use a sequence-to-sequence conversion model as an updated reconstruction network, use a multi-scale feature sequence in sequence form as guidance information, and reconstruct a preset query word sequence in combination with the guidance information to obtain a reconstructed feature sequence, and then implement anomaly detection based on the reconstructed feature sequence.

[0074] In some embodiments, the updated reconstruction network can be implemented by a neural network with an encoder-decoder (Transformer) structure. It should be noted that the updated reconstruction network has the same structure as the reconstruction network trained with the positive sample set, but the network parameters are different. In some embodiments, if the multi-scale feature sequence f is directly reconstructed by the reconstruction network with a Transformer structure, s After reconstruction, a single-input and single-output reconstructed feature sequence is obtained. in, R represents the reconstruction operation. However, this single-input and single-output method easily leads to the reconstruction network being able to learn only simple identity mappings, and it is difficult to understand the task objectives to be learned from the input samples. In this way, regardless of whether the input is a normal sample or an abnormal sample, the model can only output a simple identity mapping of the input sample, and cannot learn the ability to identify abnormal images. Therefore, the embodiment of the present disclosure introduces a learnable query word sequence, and uses the query word sequence and the multi-scale feature sequence as the dual input of the reconstruction network to perform feature reconstruction, and obtain a reconstructed feature sequence.

[0075] In some embodiments, the reconstructed feature sequence obtained by the dual-input single-output method in the embodiments of the present disclosure can be expressed as Wherein, q represents a preset query word sequence, and the preset query word sequence has the same dimension as the multi-scale feature sequence.

[0076] In the disclosed embodiment, the query word sequence is a learnable vector sequence. The electronic device can initialize the vectors in the query word sequence during the startup phase of the model training of the reconstruction network, and iteratively update the initial query word sequence during the training of the reconstruction network, and finally obtain a preset query word sequence.

[0077] In the disclosed embodiment, the electronic device encodes the multi-scale feature sequence based on the encoder part in the reconstruction network of the Transformer structure to obtain the encoding sequence. The electronic device can use the encoding sequence as guidance information for the decoder part in the reconstruction network to guide the decoding process of the encoder, so as to decode and reconstruct the preset query word sequence through the decoder based on the encoding sequence to obtain the reconstructed feature sequence.

[0078] In some embodiments, the reconstruction network of the Transformer structure can be as follows Figure 5 As shown, it includes an encoder obtained by stacking N encoding modules and a decoder obtained by stacking N decoding modules. Each of the N encoding modules includes a multi-head self-attention (Multi-head Attention) layer, a feed forward network (Feed Forward Network, FFN) and a residual and normalization (ADD & Normalize) layer. Exemplarily, the feed forward network can be a fully connected neural network.

[0079] For example, Figure 5 As shown, the electronic device will Figure 4 The multi-scale feature sequence obtained in the above is input into the encoder, and is converted into Q vector, K vector and V vector for attention calculation through three preset weight matrices, and then the attention value is calculated according to the Q vector, K vector and V vector through the multi-head self-attention layer, and the residual calculation and normalization processing are performed according to the calculated attention value and the original Q vector, K vector and V vector through the residual and normalization layer, and the processing result is linearly and nonlinearly transformed through the feedforward network, and then the residual and normalization layer is passed to obtain the encoding sequence corresponding to the multi-scale feature sequence, and the encoding sequence is input into the decoder. Then, in the decoder, the attention value calculation and residual normalization processing are performed on the preset query word sequence through the multi-head self-attention layer and the residual and normalization layer, and the encoding sequence output by the encoder is merged with the processing result of the preset query word sequence, and the attention value calculation is performed again to obtain the merged attention value, and the residual and normalization processing is performed in combination with the processing result of the preset query word sequence and the merged attention value, and then the reconstructed feature sequence is obtained through the feedforward network and the residual normalization network.

[0080] In some embodiments, Figure 5 As shown, the electronic device can also use the permutation invariance of the Transformer to add position information to each layer of the multi-head self-attention layer. For example, it can be sinusoidal position information, so as to further improve the semantic understanding ability of the reconstruction network through position information. The specific selection is made according to the actual situation, and the embodiments of the present disclosure are not limited.

[0081] Here, the electronic device uses the multi-scale feature sequence extracted from the image as guiding information to perform feature reconstruction of the learnable query word sequence in a dual-input manner, which allows the reconstruction network to combine the semantics in the query word sequence and perform decoding and reconstruction based on deep semantic understanding, thereby improving the accuracy of the reconstructed features and further improving the accuracy of anomaly detection.

[0082] The above-mentioned process of S104-S105 can be carried out by Figure 6 Indicates that Figure 6 As shown, the electronic device can extract and fuse features of different scales on the target image through the updated feature extraction network to obtain multi-scale features; perform shape transformation on the multi-scale features to obtain a multi-scale feature sequence, and then reconstruct the multi-scale feature sequence and the initial query word sequence through the updated reconstruction network to obtain a reconstructed feature sequence. Furthermore, the electronic device can perform shape transformation on the reconstructed feature sequence in sequence form to obtain a reconstructed feature.

[0083] S106: Determine the feature difference at each pixel position of the target image according to the reconstructed feature sequence and the multi-scale feature sequence.

[0084] In the disclosed embodiment, the multi-scale feature sequence represents the original features in the target image, and the reconstructed feature sequence represents the features reconstructed based on the original features. The electronic device can determine the feature difference sequence of the target image by comparing the difference between the reconstructed feature sequence and the multi-scale feature sequence.

[0085] In the embodiment of the present disclosure, each feature difference in the feature difference sequence may include corresponding pixel position information and difference value in the target image. The electronic device may determine the difference value at each pixel position in the target image through the feature difference, and characterize the reconstruction effect of the reconstruction network on each pixel point of the target image through the feature difference.

[0086] In some embodiments, the electronic device may calculate the feature difference between the reconstructed feature and the multi-scale feature. For example, the electronic device may calculate the feature difference between the reconstructed feature and the multi-scale feature by formula (2), as follows:

[0087]

[0088] In formula (2), f(u,i) is a multi-scale feature. is the reconstructed feature; u is the index coordinate of the spatial position, which can be, for example, the coordinate of each position in the feature map of the multi-scale feature or the reconstructed feature in size H×W. i is the index of the feature channel. The electronic device can perform a position-wise subtraction between the multi-scale feature and the reconstructed feature by using formula (2) to obtain the feature difference d(u,i) at each position.

[0089] In some embodiments, the electronic device may also perform difference calculation based on the multi-scale feature sequence and the reconstructed feature sequence to obtain a feature difference sequence.

[0090] S107: Determine an abnormality score at each pixel position based on the feature difference, and draw an abnormality detection image corresponding to the target image based on the abnormality score.

[0091] In the disclosed embodiment, since the reconstruction network has the characteristic that the reconstructed features of normal images or normal pixels are close to the original features, and the reconstructed features of abnormal images or abnormal pixels are far away from the original features, the electronic device can determine the abnormality score at each pixel position based on the reconstruction effect at each pixel position represented by the feature difference, and the abnormality score is used to characterize whether the corresponding pixel is an abnormal pixel. Based on the abnormality score, the electronic device distinguishes and draws the normal pixels and the abnormal pixels respectively, and obtains the abnormality detection image corresponding to the target image. In this way, the abnormal area can be automatically marked in the abnormality detection image.

[0092] In some embodiments, the electronic device may compare the abnormality score of each pixel position with a preset score interval, and determine the target pixel value corresponding to each pixel position according to the target score interval in which the abnormality score falls and the correspondence between the preset score interval and the preset pixel value. The electronic device draws an abnormality detection image according to the target pixel value corresponding to each pixel position.

[0093] Exemplarily, when the abnormality score at each pixel position is less than or equal to the preset score threshold, the electronic device draws the pixel value corresponding to the pixel position as the first value; when the abnormality score at each pixel position is greater than the preset score threshold, the electronic device draws the pixel value corresponding to the pixel position as the second value; until the pixel position of the target image is drawn, an abnormality detection image is obtained. For example, for a target image A, the electronic device can obtain the abnormality score of the target image A, and draw an abnormality detection image A' based on the abnormality score, so as to use the abnormality detection image A' to characterize whether the target image A is a normal image or an abnormal image. Exemplarily, Figure 7FIG. 1 is a schematic diagram of an abnormality detection image A′ obtained after abnormality detection is performed on a target image A according to an exemplary embodiment of the present disclosure; Figure 7 As shown, in the case where the target image A is a toothbrush image, the drawn abnormality detection image is a toothbrush image A', and, as shown in image A, in the case where the bristles at the tip of the brush head are abnormal, in the drawn toothbrush image A', the pixel color (pixel value) of the area 7-10 where the bristles at the tip of the brush head are located is different from the pixel color of the area 7-20 where the bristles in other parts are located, thereby characterizing the target image A as an abnormal image.

[0094] In some embodiments, the above S101 may be implemented through S1011-S1013; Figure 8 is an optional flow chart of the anomaly detection method provided by the embodiment of the present disclosure, which is combined with Figure 8 The steps shown are explained.

[0095] S1011. Use an initial detection network to detect positive samples in the first sample set to obtain a first feature difference sequence corresponding to the positive samples.

[0096] In the disclosed embodiment, for each positive sample, the electronic device can obtain the multi-scale feature sequence of the positive sample through the initial detection network, and reconstruct it according to the multi-scale feature sequence and the preset query word sequence to obtain the reconstructed feature sequence of the positive sample, and determine the feature difference sequence of the positive sample by comparing the difference between the reconstructed feature sequence and the multi-scale feature sequence, hereinafter referred to as the first feature difference sequence. It should be noted that the network structure of the initial detection network and the updated detection network is the same, but the network parameters are different, so the specific process of the electronic device obtaining the first feature difference sequence corresponding to the positive sample is the same as the above-mentioned S104-S106 part, and will not be described in detail here.

[0097] S1012. Determine the training loss corresponding to the positive sample according to the first feature difference sequence and the normal loss function; wherein the normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is closer to the positive sample.

[0098] In the disclosed embodiment, when the electronic device obtains the first feature difference sequence, it can use a normal loss function to calculate the training loss corresponding to the completely positive sample.

[0099] In some embodiments, the normal loss function may include a regression loss function; for example, it may be a smooth L1 loss, an MSE loss, etc., which is selected according to the actual situation and is not limited in the embodiments of the present disclosure. For example, the electronic device may use the smooth L1 loss as a normal loss function to calculate the training loss corresponding to the positive sample, as shown in formula (3), as follows:

[0100]

[0101] In formula (3), represents the first characteristic difference or the first characteristic difference sequence, L nor Represents the training loss corresponding to the positive sample.

[0102] In some embodiments, Fig. 9 It shows that in the case of a positive sample, the training loss corresponding to the positive sample is obtained through the first feature difference sequence of the positive sample.

[0103] It is understandable that the electronic device can evaluate the reconstruction ability of the initial detection network for positive samples by calculating the gap between the original features and the reconstructed features through a normal loss function, so that the initial detection network can support training using positive samples.

[0104] S1013. Based on the training loss, the initial detection network is trained and adjusted until the final loss is less than a preset loss threshold, thereby obtaining the detection network.

[0105] In the disclosed embodiment, when the electronic device obtains the training loss corresponding to the positive sample, it can adjust the network parameters of the initial detection network based on the training loss corresponding to the positive sample to complete the current round of training process using the positive sample. The electronic device can continue to use the positive samples in the positive sample set to iteratively train the initial detection network until the final loss is less than the preset loss threshold, thereby obtaining the detection network.

[0106] In the embodiment of the present disclosure, the preset loss threshold can be set according to actual needs, and the embodiment of the present disclosure is not limited to this.

[0107] In some embodiments, the initial detection network can be composed of a pre-trained feature extraction network and an initial reconstruction network; based on this, the electronic device can train the initial reconstruction network according to the obtained first feature difference sequence and the normal loss function, and obtain the reconstruction network when the final loss is less than a preset loss threshold, thereby obtaining a detection network composed of the pre-trained feature extraction network and the obtained reconstruction network.

[0108] In other embodiments, the initial detection network may be composed of an initial feature extraction network and an initial reconstruction network. The electronic device may calculate the training loss corresponding to the initial feature extraction network based on the multi-scale features extracted by the initial feature extraction network and the loss function corresponding to the initial feature extraction network. At the same time, the first feature difference sequence is obtained based on the obtained multi-scale features and the reconstructed features, and the training loss corresponding to the initial reconstruction network is determined based on the first feature difference sequence and the normal loss function corresponding to the initial reconstruction network. The initial feature extraction network is adjusted using the training loss corresponding to the initial feature extraction network, and the initial reconstruction network is trained and adjusted using the training loss corresponding to the initial reconstruction network, until the final loss corresponding to the initial feature extraction network is less than the corresponding preset loss threshold, and the final loss corresponding to the initial reconstruction network is less than the corresponding preset loss threshold, a feature extraction network and a reconstruction network are obtained, thereby obtaining a detection network composed of a feature extraction network and a reconstruction network. In some embodiments, the electronic device may also train the initial feature extraction network and the initial reconstruction network separately, and the training method is the same as the above method, which will not be described in detail here.

[0109] In some embodiments, after the above S101, S201-S203 may also be performed; Fig.10 is an optional flow chart of the anomaly detection method provided by the embodiment of the present disclosure, which is combined with Fig.10 The steps shown are explained.

[0110] S201, using a detection network to perform anomaly detection on each image to be detected, and obtaining an anomaly score at each pixel position of each image to be detected.

[0111] In the disclosed embodiment, for each image to be detected, the electronic device can use a detection network to perform feature processing of different scales on the image to be detected to obtain a multi-scale feature sequence, and reconstruct the obtained multi-scale feature sequence and a preset query word sequence to obtain a reconstructed feature sequence, and determine the feature difference at each pixel position of the image to be detected based on the reconstructed feature sequence and the obtained multi-scale feature sequence, and determine the abnormality score at each pixel position of the image to be detected based on the feature difference.

[0112] S202: based on the anomaly score, draw an anomaly detection image corresponding to each image to be detected.

[0113] S203: Obtain a normal image set and an abnormal image set from a plurality of images to be detected according to the abnormal detection image.

[0114] In the disclosed embodiment, for each image to be detected, the electronic device can draw an abnormality detection image corresponding to the image to be detected based on the abnormality score at each pixel position of the image to be detected and a preset score threshold, and determine whether the image to be detected is an abnormal image or a normal image based on the corresponding abnormality detection image; in this way, the electronic device can divide multiple detected images to be detected into a normal image set and an abnormal image set, thereby obtaining a normal image set containing at least one normal image, and an abnormal image set containing at least one abnormal image.

[0115] In some embodiments of the present disclosure, the electronic device may, when each image to be detected has an abnormal detection image representation corresponding to it and the image to be detected is an abnormal image, treat the image to be detected as an abnormal image, and after traversing multiple images to be detected, obtain an abnormal image set including at least one abnormal image; and, when each image to be detected has an abnormal detection image representation corresponding to it and the image to be detected is a normal image, treat the image to be detected as a normal image, and after traversing multiple images to be detected, obtain a normal image set including at least one normal image; in this way, an abnormal image set and a normal image set may be obtained.

[0116] In some embodiments of the present disclosure, the electronic device may stop detecting the images to be detected when the number of detected images to be detected reaches a preset number, and divide the multiple images to be detected into a normal image set and an abnormal image set according to the abnormal detection image corresponding to each of the multiple images to be detected that have been detected; in this way, a sufficient number of normal images and abnormal images can be obtained, which is conducive to the subsequent updated detection network with higher detection accuracy. In some embodiments of the present disclosure, the electronic device may also stop detecting the images to be detected when the number of abnormal images in the obtained abnormal image set reaches a preset number, or when the number of normal images in the obtained normal image set reaches a preset number, and divide the multiple images to be detected into a normal image set and an abnormal image set according to the abnormal detection image corresponding to each of the multiple images to be detected that have been detected; in this way, a sufficient number of abnormal images and normal images can be obtained, which is conducive to the subsequent updated detection network with higher detection accuracy.

[0117] In some embodiments of the present disclosure, the electronic device may also stop detecting the image to be detected, and divide the multiple images to be detected into a normal image set and an abnormal image set according to the abnormal detection image corresponding to each image to be detected in the detected multiple images to be detected into an abnormal image set and an abnormal image set according to the abnormal detection image corresponding to each image to be detected in the detected multiple images to be detected; in this way, a sufficient number of normal suspicious images or abnormal suspicious images can be obtained, which is conducive to obtaining an updated detection network with higher detection accuracy in the subsequent detection. It should be noted that the first preset numerical range of the preset abnormal threshold and the second preset numerical range of the preset abnormal threshold are both ranges composed of numerical values ​​close to the preset abnormal threshold, and can be set according to actual needs. In some embodiments of the present disclosure, the first maximum anomaly score corresponding to the abnormal image is usually lower than the preset anomaly threshold, and the second maximum anomaly score corresponding to the normal image is usually higher than the preset anomaly threshold. For example, when the preset anomaly threshold is 5, the first preset numerical range can be [4.5, 5], and the second preset numerical range can be (5, 5.5).

[0118] In some embodiments of the present disclosure, the electronic device may also stop detecting the image to be detected when the number of abnormal suspicious images in the obtained abnormal image set reaches a preset number; and when the number of normal suspicious images in the obtained normal image set reaches a preset number, and divide the multiple images to be detected into a normal image set and an abnormal image set according to the abnormal detection image corresponding to each image to be detected in the multiple images to be detected that have been detected; in this way, a sufficient number of normal suspicious images and abnormal suspicious images can be obtained, which is conducive to obtaining an updated detection network with higher detection accuracy in the subsequent detection.

[0119] In some embodiments, obtaining the second sample set based on the detected normal image set and abnormal image set in the above S102 can be implemented through S1021-S1022.

[0120] S1021. Determine an image to be verified from the detected normal image set and abnormal image set.

[0121] S1022: Verify the image to be verified to obtain a second sample set.

[0122] In the disclosed embodiment, the electronic device can determine the images to be verified based on all normal images and abnormal images contained in the obtained normal image set and abnormal image set for verification, and obtain a second sample set consisting of positive samples and negative samples based on the verification results.

[0123] In the embodiments of the present disclosure, the electronic device may automatically verify the image to be verified, or may verify the image to be verified based on a verification operation received from the user, and the embodiments of the present disclosure are not limited to this.

[0124] In some embodiments, the electronic device may verify all normal images and abnormal images included in the normal image set and the abnormal image set as images to be verified.

[0125] In some embodiments, the electronic device may also select some normal images and some abnormal images from the normal image set and the abnormal image set as images to be verified. For example, the above S1022 may be implemented using S301-S304:

[0126] S301. Determine, from each abnormal image in the abnormal image set, a first maximum abnormal score that is the largest among the abnormal scores at each pixel position.

[0127] When the electronic device obtains the abnormal image set, for each abnormal image in the abnormal image set, a maximum abnormality score may be determined from the abnormality scores of all pixels in the abnormal image as a first maximum abnormality score.

[0128] S302 . Determine, from each normal image in the normal image set, a second maximum abnormality score that is the largest among the abnormality scores at each pixel position.

[0129] When the electronic device obtains the normal image set, for each normal image in the normal image set, it can determine a maximum abnormality score from the abnormality scores of all pixels in the normal image as the second maximum abnormality score.

[0130] S303: When the first maximum anomaly score belongs to a first preset value range of a preset anomaly threshold, determine that the abnormal image corresponding to the first maximum anomaly score belongs to an image to be verified.

[0131] For an abnormal image, when the electronic device selects a first maximum abnormality score, it can determine whether the selected first maximum abnormality score is within a first preset numerical range of a preset abnormality threshold, and when the selected first maximum abnormality score is within the first preset numerical range of the preset abnormality threshold, the abnormal image is used as an image to be verified.

[0132] In some embodiments of the present disclosure, the first preset numerical range of the preset abnormality threshold is a range consisting of numerical values ​​close to the preset abnormality threshold, which can be set according to actual needs, and the embodiments of the present disclosure are not limited to this. In some embodiments of the present disclosure, the first maximum abnormality score corresponding to the abnormal image is usually lower than the preset abnormality threshold. For example, when the preset abnormality threshold is 5, the first preset numerical range can be [4.5,5); then, when the first maximum abnormality score determined from the abnormality scores of all pixels of an abnormal image B is 4.5, it means that the first maximum abnormality score corresponding to the abnormal image B belongs to the first preset numerical range [4.5,5), and the abnormal image B can be used as the image to be verified.

[0133] S304: When the second maximum anomaly score belongs to a second preset value range of the preset anomaly threshold, determine that the normal image corresponding to the second maximum anomaly score belongs to the image to be verified.

[0134] When the electronic device obtains a normal image set, for each normal image in the normal image set, it can determine a maximum abnormality score from the abnormality scores of all pixels in the normal image as the second maximum abnormality score, and determine whether the selected second maximum abnormality score is within a second preset numerical range of a preset abnormality threshold, and when the selected second maximum abnormality score is within the second preset numerical range of the preset abnormality threshold, the normal image is used as an image to be verified.

[0135] In some embodiments of the present disclosure, the second preset numerical range of the preset abnormality threshold is also a range composed of numerical values ​​close to the preset abnormality threshold, which can be set according to actual needs. In some embodiments of the present disclosure, the second maximum abnormality score corresponding to the normal image is usually higher than the preset abnormality threshold. For example, when the preset abnormality threshold is 5, the second preset numerical range can be (5,5.5]; then when the second maximum abnormality score determined from the abnormality scores of all pixels of a normal image C is 5.1, it means that the second maximum abnormality score corresponding to the normal image C belongs to the second preset numerical range (5,5.5], and the normal image C can be used as the image to be verified.

[0136] In some embodiments, the above S1022 may be implemented through S401-S404; Fig.11 is an optional flow chart of the anomaly detection method provided by the embodiment of the present disclosure, which is combined with Fig.11 The steps shown are explained.

[0137] S401. Verify at least one abnormal image and at least one normal image respectively to obtain respective verification results; the images to be verified include: at least one abnormal image and at least one normal image.

[0138] In the disclosed embodiment, the electronic device can verify each image in the abnormal image set and the normal image set to determine whether the abnormal detection result of the image is correct, and obtain the verification result of the image, and the verification result represents whether the detection result of the image is correct or wrong.

[0139] S402: taking the abnormal images whose verification results are correct as negative samples, and taking the normal images whose verification results are correct as positive samples.

[0140] In the disclosed embodiment, for abnormal images whose verification results are correct, the electronic device sets labeling information for the abnormal images to characterize that the abnormal images are negative samples, and obtains negative samples; for normal images whose verification results are correct, the electronic device sets labeling information for the normal images to characterize that the normal images are positive samples, and obtains positive samples for subsequent updating and training of the detection network.

[0141] S403, correctly labeling abnormal images with incorrect verification result representations and normal images with incorrect verification result representations to obtain positive samples and negative samples.

[0142] In the disclosed embodiment, for an abnormal image with an incorrect verification result, the electronic device will determine whether the abnormal image is an abnormal image or a normal image, and annotate the abnormal image with correct annotation information based on the determination result; and, for a normal image with an incorrect verification result, the electronic device will determine whether the normal image is an abnormal image or a normal image, and annotate the normal image with correct annotation information based on the determination result; after all abnormal images with incorrect verification results and all normal images with incorrect verification results are correctly annotated, the electronic device can obtain positive samples and negative samples based on the correctly annotated annotation information.

[0143] S404: Determine the set of negative samples and positive samples as a second sample set.

[0144] In the disclosed embodiment, the electronic device may use all the obtained negative samples and positive samples as the second sample set for updating and training the detection network.

[0145] In some embodiments, the above S103 may be implemented through S1031-S1033; Fig.12 is an optional flow chart of the anomaly detection method provided by the embodiment of the present disclosure, which is combined with Fig.12 The steps shown are explained.

[0146] S1031. When using the positive samples in the second sample set to train the detection network, the detection network is used to obtain a first feature difference sequence corresponding to the positive sample, and a first training loss corresponding to the positive sample is determined based on the first feature difference sequence and a normal loss function; the normal loss function characterizes the reconstructed feature sequence corresponding to the positive sample to bring it closer to the positive sample.

[0147] In the embodiment of the present disclosure, the electronic device can use the positive samples in the second sample set to train the detection network, and can Figure 6 The method in performs feature processing and reconstruction on the currently acquired positive sample, and uses the above formula (2) to calculate the feature difference, etc., to obtain the first feature difference or first feature difference sequence corresponding to the positive sample, and then uses the normal loss function to calculate the first training loss corresponding to the positive sample, so as to adjust the network parameters of the detection network. It should be noted that the process of S1031 is the same as the content of the above S1011-S1012 parts, and will not be described here.

[0148] S1032. When using negative samples in the second sample set to train the detection network, the detection network is used to obtain a second feature difference sequence corresponding to the negative sample, and a second training loss corresponding to the negative sample is determined based on the second feature difference sequence, the true value of the preset negative sample, and the abnormal loss function.

[0149] In an embodiment of the present disclosure, the electronic device can obtain negative samples from the second sample set, and the true value of the preset negative sample corresponding to the negative sample, when using the negative samples in the second sample set to train the detection network. Exemplarily, the true value of the preset negative sample can be annotation information in the form of a pixel-level label or an image-level label, wherein the pixel-level label can be annotation information for each pixel in the negative sample, used to label each pixel as a normal pixel or an abnormal pixel; the image-level label can be annotation information for the entire image, used to mark whether there is an abnormality in the entire image. Exemplarily, the image-level label corresponding to an image containing some abnormal pixels can be a label that characterizes the abnormality of the entire image.

[0150] In the implementation of the present disclosure, the electronic device is Figure 6 The method performs feature processing and reconstruction on the negative sample, and uses the above formula (2) to perform difference calculation and other processing to obtain a second feature difference or a second feature difference sequence corresponding to the negative sample; then, the electronic device calculates the difference between the second feature difference or the second feature difference sequence and the true value of the preset negative sample through an abnormal loss function, and obtains a second training loss corresponding to the negative sample, which is used to adjust the network parameters of the detection network.

[0151] S1033. Based on the first training loss and the second training loss, the detection network is trained and adjusted respectively until the final loss is less than a preset loss threshold, thereby obtaining an updated detection network; wherein the abnormal loss function represents that the reconstructed feature sequence corresponding to the abnormal part in the negative sample is pushed away from the abnormal part, and the reconstructed feature sequence corresponding to the normal part in the negative sample is brought closer to the normal part.

[0152] In the disclosed embodiment, the electronic device adjusts the network parameters of the detection network based on the first training loss corresponding to the positive sample or the second training loss corresponding to the negative sample, and completes the current round of training process using the positive sample or the negative sample. Afterwards, the electronic device can continue to use the positive samples and negative samples in the incremental samples to iteratively train the detection network until the preset training conditions are met, such as when the final loss is less than the preset loss threshold, to obtain an updated detection network.

[0153] It should be noted that, in the disclosed embodiment, as described in the above S1013 section, the detection network may be composed of a pre-trained feature extraction network and a reconstruction network, and the electronic device may only train the reconstruction network to obtain an updated detection network composed of a pre-trained feature extraction network and an updated reconstruction network; and, the detection network may also be composed of a feature extraction network and a reconstruction network, and the electronic device may simultaneously train the feature extraction network and the reconstruction network, or train the feature extraction network and the reconstruction network separately to obtain an updated detection network composed of an updated feature extraction network and an updated reconstruction network. The process of the electronic device using the second sample set to train the detection network is the same as the process in the above S1013 section, except that the samples used are different and the training losses corresponding to the obtained samples are different; this is not elaborated in the disclosed embodiment.

[0154] In some embodiments, for different types of true values ​​of preset negative samples, such as pixel-level labels or image-level labels, the electronic device may calculate the second training loss corresponding to the negative sample by using different abnormal loss functions. It should be noted that for different types of true values ​​of preset negative samples, the overall characteristics that the abnormal loss function needs to meet are: the reconstructed feature sequence corresponding to the abnormal part in the negative sample is pushed away from the abnormal part, and the reconstructed feature sequence corresponding to the normal part in the negative sample is pulled closer to the normal part.

[0155] In some embodiments, the true value of the preset negative sample includes: a pixel-level true value, that is, a pixel-level label; the abnormal loss function includes: a pixel-level loss function. The electronic device trains and adjusts the detection network based on the second feature difference sequence of the negative sample, the preset true value of the negative sample, and the abnormal loss function until the updated detection network with a final loss less than the preset loss threshold is obtained. The process may include:

[0156] (1) The second feature difference sequence of the negative sample is scalarized in the channel dimension to obtain a feature difference measure at each pixel position; here, when the true value of the negative sample is a preset pixel-level true value, taking the calculation of the abnormal loss function through the second feature difference sequence as an example, the electronic device can convert the feature difference vector representing each pixel position in the second feature difference sequence into a scalar in the channel dimension, so that each pixel position corresponds to only one scalar feature difference measure; (2) Based on the feature difference measure, the pixel-level true value and the pixel-level loss function, the detection network is trained and adjusted until an updated detection network is obtained whose final loss is less than a preset loss threshold.

[0157] In some embodiments, the pixel-level loss function is a loss function with the above-mentioned "zoom-in-and-pull-away" characteristics, and its design goal can be: using the pixel-level loss function, the feature difference measure at the corresponding position of the normal pixel points in the negative sample is reduced to bring the reconstructed features of the normal pixel points closer to the original features, such as the distance between multi-scale features; at the same time, the feature difference measure at the corresponding position of the abnormal pixel points in the negative sample is increased, that is, the distance between the reconstructed features of the abnormal pixel points and the original features is pushed away.

[0158] In some embodiments, the pixel-level loss function includes: a normal pixel loss part and an abnormal pixel loss part. When a pixel point represented by the true value of the pixel level corresponding to a negative sample is a normal pixel point, the electronic device can perform a weighted average of the feature difference measurement of the normal pixel point based on the normal pixel loss part to obtain the normal pixel point loss; when a pixel point represented by the true value of the pixel level corresponding to a negative sample is an abnormal pixel point, the electronic device can perform a weighted average of the feature difference measurement of the abnormal pixel point based on the abnormal pixel loss part to obtain the abnormal pixel point loss. Among them, the normal pixel point loss is positively correlated with the feature difference measurement of the normal pixel point, and the abnormal pixel point loss is negatively correlated with the feature difference measurement of the abnormal pixel point, thereby achieving the effect of shortening the distance between the reconstructed features of the normal pixel point and the original feature, and pushing away the distance between the reconstructed features of the abnormal pixel point and the original feature.

[0159] It is understandable that electronic devices can use pixel-level loss functions to evaluate the detection network's ability to reconstruct negative samples with pixel-level labels, so that the detection network's reconstructed features for normal pixels are close to the original features, and the reconstructed features for abnormal pixels are far from the original features, thereby being compatible with the training of negative samples with pixel-level labels, improving the flexibility and accuracy of the detection network in training with different types of samples, and thereby improving the flexibility and accuracy of anomaly detection based on the detection network.

[0160] In some embodiments of the present disclosure, when the electronic device scalarizes the feature difference vector at each pixel position, it averages the feature difference vectors corresponding to different feature channels equally. In practical applications, different feature channels may contribute differently to anomalies. Therefore, the electronic device may also perform weight analysis on the feature difference vector corresponding to each feature channel through a classification network to calculate the pixel-level loss function in combination with different weights to further improve accuracy. In this case, the electronic device trains and adjusts the detection network based on the second feature difference sequence, the true value of the preset negative sample, and the anomaly loss function until the final loss is less than the preset loss threshold. The process of the updated detection network may include:

[0161] (3) Using a classification network, classify the pixel positions of the second feature difference sequence to obtain the sample classification probability at each pixel position; (4) Based on the sample classification probability at each pixel position, the true value at the pixel level and the classification loss function, obtain the current classification loss; (5) Based on the feature difference metric, the true value at the pixel level and the pixel level loss function, obtain the first current loss; (6) Based on the current classification loss and the first current loss, train and adjust the detection network and the classification network until an updated classification network and an updated detection network are obtained whose final loss is less than a preset loss threshold.

[0162] In some embodiments, for the second feature difference d(u,i) of H×W×C dimensions, each feature difference vector in the second feature difference can be represented by a C-dimensional d(u). The electronic device can classify each d(u) by formula (4) to obtain the probability that each d(u) is abnormal or defective as the sample classification probability at the corresponding pixel position u, thereby obtaining the sample classification probability at each pixel position, as follows:

[0163] p(u)=C(d(u)) (4)

[0164] In formula (4), p(u) is the sample classification probability at pixel position u. In this way, for each C-dimensional feature difference vector, the electronic device can use the classification network to combine the feature differences corresponding to each feature channel in the C-dimensional channel to perform classification prediction and obtain the sample classification probability at each pixel position.

[0165] In some embodiments, the classification network may be FFN or other network models with classification functions. The specific selection is made according to actual conditions and is not limited in the embodiments of the present disclosure.

[0166] It can be understood that by introducing a classification network to perform classification prediction on the second feature difference sequence, the difference vectors of each dimension in each feature difference vector can be combined to obtain the sample classification probability at each pixel position, and the first current loss is calculated based on the sample classification probability weight, thereby further improving the accuracy of training the detection network using the first current loss, and then improving the accuracy of anomaly detection using the updated detection network.

[0167] In some embodiments, the true value of the preset negative sample includes: an image-level true value, that is, an image-level pixel label; the abnormal loss function includes: an image-level loss function. The electronic device trains and adjusts the detection network based on the second feature difference sequence, the true value of the preset negative sample, and the abnormal loss function until the updated detection network with a final loss less than the preset loss threshold is obtained. The process may include:

[0168] (7) performing channel dimension scalarization processing on the second feature difference sequence to obtain a feature difference measure at each pixel position; (8) determining the K largest feature difference measures with the largest values ​​from the feature difference measures at each pixel position; K is a positive integer greater than 1; (9) averaging the K largest feature difference measures to obtain a feature average difference measure; (10) training and adjusting the detection network based on the feature average difference measure, the true value at the image level, and the image level loss function until an updated detection network is obtained whose final loss is less than a preset loss threshold.

[0169] In some embodiments, the image-level loss function includes: a normal image loss part and an abnormal image loss part. When the image-level true value representation image corresponding to a negative sample is a normal image, the electronic device determines the feature average difference metric as the normal image loss based on the normal image loss part; when the image-level true value representation image corresponding to a negative sample is a negative image, the electronic device obtains the abnormal image loss based on the abnormal image loss part and the feature average difference metric. Among them, the feature average difference metric of the normal image is positively correlated with the normal image loss, and the feature average difference metric of the abnormal image is negatively correlated with the abnormal image loss. The electronic device determines the second current loss based on the normal image loss or the abnormal image loss; based on the second current loss and the preset loss threshold, the detection network is trained and adjusted until an updated detection network is obtained whose final loss is less than the preset loss threshold.

[0170] In some embodiments, the image-level loss function is a loss function having the above-mentioned "zoom-in-and-pull-away" characteristics, and its design goal may be: using the image-level loss function, the average difference metric of features corresponding to the normal image is reduced to bring the distance between the reconstructed features and the original features of the normal image closer; at the same time, the average difference metric of features corresponding to the abnormal image is increased, that is, the distance between the reconstructed features and the original features of the abnormal image is pushed away.

[0171] It can be understood that electronic devices can evaluate the detection network's ability to reconstruct negative samples with image-level labels through image-level loss functions, so that the detection network's reconstructed features for normal images are close to the original features, and the reconstructed features for negative samples are far from the original features, thereby being compatible with the training of negative samples with image-level labels, improving the flexibility and accuracy of the updated detection network in training with different types of samples, and thereby improving the flexibility and accuracy of anomaly detection based on the updated detection network.

[0172] In some embodiments, when the detection network or the updated detection network is not trained using a classification network, the method for the electronic device to calculate the anomaly score at each pixel position can be: channel averaging the feature differences at each pixel position to obtain an intermediate anomaly score at each pixel position; and using the intermediate anomaly score as the anomaly score.

[0173] In some embodiments, the electronic device may perform channel averaging on the feature difference at each pixel position in the image to be detected or the target image to obtain an intermediate anomaly score at each pixel position, and use the intermediate anomaly score as the anomaly score, as shown in formula (5), as follows:

[0174]

[0175] In formula (5), d'(u,i) is the feature difference at each pixel position, C is the number of feature channels corresponding to each feature difference in the feature difference sequence, and s(u) is the intermediate anomaly score obtained by channel averaging the feature differences at each pixel position, that is, the anomaly score at each pixel position in the image to be detected or the target image output by the detection network without the classification network.

[0176] In some embodiments, when a detection network is obtained by training a classification network, a method for an electronic device to calculate an anomaly score at each pixel position may be: channel averaging the feature differences at each pixel position to obtain an intermediate anomaly score at each pixel position; using a classification network to classify the feature differences at each pixel position at the pixel position to obtain a classification probability; and multiplying the classification probability by the intermediate anomaly score to obtain an anomaly score at each pixel position.

[0177] In some embodiments, the electronic device uses the classification probability as a weight and weights it with the intermediate anomaly score to obtain an anomaly score at each pixel position. For example, as shown in formula (6):

[0178]

[0179] In the disclosed embodiment, on the basis of adopting positive samples to obtain a trained detection network, the trained detection network is further trained using incremental samples including negative samples and positive samples until a trained and updated detection network is obtained, so that the detection network can be compatible with both positive sample learning and incremental abnormal sample learning, thereby improving the compatibility and flexibility of the detection network.

[0180] Fig.13 is a flow chart of the updated detection network provided by the embodiment of the present disclosure. Fig.13 , the technical solution of the embodiment of the present disclosure is explained through a detailed embodiment.

[0181] During the cold start phase, the electronic device continuously trains the initial detection network using a preset positive sample set until a trained detection network is obtained. In the deployment incremental training phase, the electronic device uses multiple images to be detected generated on the production line as production line data sets, inputs them into the detection network trained in the cold start phase, and collects abnormal detection images corresponding to each image to be detected output by the detection network, uses the abnormal detection images as the collected detection results, and determines whether the corresponding image to be detected is a normal image or an abnormal image based on the collected detection results, thereby dividing the multiple images to be detected that have been detected into a normal image set and an abnormal image set, and uses the normal image set as a normal sample, and uses the abnormal image set as an abnormal sample; thereafter, both the normal samples and the abnormal samples are verified, and the verified normal samples and abnormal samples with labeled information are obtained, and the normal samples with labeled information (positive samples) and the abnormal samples with labeled information (negative samples) are used together as incremental data sets (incremental sample sets), and the incremental data sets are used to continue to train the detection network trained in the cold start phase until a trained updated detection network is obtained, and the updated detection network is used to update the detection network obtained in the cold start phase, so that the updated detection network is used to continue to detect the images generated on the production line.

[0182] The anomaly detection method provided in the embodiments of the present disclosure can be applied to intelligent artificial intelligence (AI) training platforms, anomaly detection, and online automatic training and improvement of anomaly detection.

[0183] The present disclosure also provides a detection device, Fig.14A schematic diagram of the structure of an abnormality detection device provided in an embodiment of the present disclosure; Fig.14 As shown, the abnormality detection device 500 includes: a first training unit 501, which is used to train the initial detection network with a first sample set to obtain the detection network; the first sample set is a positive sample set; an acquisition unit 502, which is used to obtain a second sample set based on the detected normal image set and abnormal image set during the abnormality detection of multiple images to be detected by the detection network, wherein the second sample set is an incremental sample set including positive samples and negative samples; wherein the negative sample is an abnormal image with abnormalities in the image; and a second training unit 503, which is used to update the detection network with the second sample set to obtain an updated detection network. .

[0184] In some embodiments of the present disclosure, the first training unit 501 is further used to use the initial detection network to detect positive samples in the first sample set to obtain a first feature difference sequence corresponding to the positive sample; determine the training loss corresponding to the positive sample based on the first feature difference sequence and a normal loss function; based on the training loss, perform training adjustment on the initial detection network until the final loss obtained is less than a preset loss threshold, thereby obtaining the detection network; wherein the normal loss function characterizes that the reconstructed feature sequence corresponding to the positive sample is closer to the positive sample.

[0185] In some embodiments of the present disclosure, the acquisition unit 502 is further used to determine the images to be verified from the detected normal image set and the abnormal image set; and verify the images to be verified to obtain the second sample set.

[0186] In some embodiments of the present disclosure, the image to be verified includes: at least one abnormal image and at least one normal image; the acquisition unit 502 is also used to verify the at least one abnormal image and the at least one normal image respectively to obtain respective verification results; the abnormal image represented by the verification result correctly is used as the negative sample, and the normal image represented by the verification result correctly is used as the positive sample; the abnormal image represented by the verification result incorrectly and the normal image represented by the verification result incorrectly are correctly labeled to obtain the positive sample and the negative sample; the set of the negative sample and the positive sample is determined as the second sample set.

[0187] In some embodiments of the present disclosure, the anomaly detection device 500 also includes: a detection unit; the detection unit is further used to use the first sample set to train the initial detection network, and after obtaining the detection network, use the detection network to perform anomaly detection on each image to be detected to obtain an anomaly score at each pixel position of each image to be detected; based on the anomaly score, draw an abnormal detection image corresponding to each image to be detected; the acquisition unit 502 is also used to obtain the normal image set and the abnormal image set in multiple images to be detected based on the abnormal detection image.

[0188] In some embodiments of the present disclosure, the acquisition unit 502 is further used to represent the abnormal detection image corresponding to each image to be detected, and when each image to be detected is an abnormal image, take each image to be detected as an abnormal image, and after traversing the multiple images to be detected, obtain the abnormal image set containing at least one abnormal image; and represent the abnormal detection image corresponding to each image to be detected, and when each image to be detected is a normal image, take each image to be detected as a normal image, and after traversing the multiple images to be detected, obtain the normal image set containing at least one normal image.

[0189] In some embodiments of the present disclosure, the acquisition unit 502 is further used to determine a first maximum abnormality score that is the largest among the abnormality scores at each pixel position from each abnormal image in the abnormal image set; determine a second maximum abnormality score that is the largest among the abnormality scores at each pixel position from each normal image in the normal image set; when the first maximum abnormality score belongs to a first preset numerical range of a preset abnormal threshold, determine that the abnormal image corresponding to the first maximum abnormality score belongs to the image to be verified; when the second maximum abnormality score belongs to a second preset numerical range of the preset abnormal threshold, determine that the normal image corresponding to the second maximum abnormality score belongs to the image to be verified.

[0190] In some embodiments of the present disclosure, the acquisition unit 502 is further configured to determine all images in the normal image set and the abnormal image set as the images to be verified.

[0191] In some embodiments of the present disclosure, the second training unit 502 is further used for, when the positive samples in the second sample set are used to train the detection network, obtaining a first feature difference sequence corresponding to the positive sample by using the detection network, and determining a first training loss corresponding to the positive sample according to the first feature difference sequence and a normal loss function; when the negative samples in the second sample set are used to train the detection network, obtaining a second feature difference sequence corresponding to the negative sample by using the detection network, and determining a second training loss corresponding to the negative sample according to the second feature difference sequence, a preset true value of the negative sample and an abnormal loss function; based on the first training loss and the second training loss, respectively training and adjusting the detection network until the final loss obtained is less than a preset loss threshold, thereby obtaining the updated detection network; wherein the normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is brought closer to the positive sample; the abnormal loss function represents that the reconstructed feature sequence corresponding to the abnormal part in the negative sample is pushed away from the abnormal part, and the reconstructed feature sequence corresponding to the normal part in the negative sample is brought closer to the normal part.

[0192] In some embodiments of the present disclosure, the updated detection network includes: an updated feature extraction network and an updated reconstruction network; the detection unit is further used to use the second sample set to update the detection network, and after obtaining the updated detection network, use the updated feature extraction network to perform feature processing of different scales on the target image to obtain a multi-scale feature sequence; use the updated reconstruction network to reconstruct the multi-scale feature sequence and a preset query word sequence to obtain a reconstructed feature sequence; determine the feature difference at each pixel position of the target image based on the reconstructed feature sequence and the multi-scale feature sequence; determine the anomaly score at each pixel position based on the feature difference, and draw the anomaly detection image corresponding to the target image based on the anomaly score.

[0193] The present disclosure also provides an electronic device, Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as Fig.15 As shown, the electronic device 2 includes: a memory 21 and a processor 22, wherein the memory 21 and the processor 22 are connected via a communication bus 23; the memory 21 is used to store executable instructions (executable computer programs); the processor 22 is used to implement the method provided by the embodiment of the present disclosure when executing the executable instructions stored in the memory 21, for example, the anomaly detection method provided by the embodiment of the present disclosure.

[0194] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for implementing the method provided by the embodiment of the present disclosure when executed by the processor 22, for example, the anomaly detection method provided by the embodiment of the present disclosure.

[0195] In some embodiments of the present disclosure, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0196] In some embodiments of the present disclosure, executable instructions may be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0197] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0198] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0199] In summary, the positive sample set is used to train the initial detection network to obtain the detection network. In the process of the detection network performing anomaly detection on multiple images to be detected, an incremental sample set including positive samples and negative samples is obtained based on the detected normal image set and abnormal image set, wherein the negative sample is an abnormal image with anomalies in the image. The incremental sample set including positive samples and negative samples is used to update the detection network to obtain an updated detection network. Therefore, the obtained updated detection network is more suitable for the detection scenario of the actual production line, and the detection accuracy of the updated detection network is higher, so that when the updated detection network is used to perform anomaly detection on the image to be detected later, the detection result obtained is more accurate. At the same time, since the incremental samples include positive samples and negative samples, the obtained updated detection network can be obtained by training with incremental samples of positive samples and negative samples, so that the detection network can be compatible with different situations such as positive samples and negative samples, thereby improving the versatility and flexibility of anomaly detection.

[0200] The above description is only an embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement and improvement made within the spirit and scope of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. An anomaly detection method, characterized in that: include: Using the first sample set to train the initial detection network to obtain a detection network; The first sample set is a positive sample set; In the process of performing anomaly detection on a plurality of images to be detected by the detection network, a second sample set is obtained based on the detected normal image set and abnormal image set, wherein the second sample set is an incremental sample set including positive samples and negative samples; wherein the negative sample is an abnormal image in which an abnormality exists; Using the second sample set, updating and training the detection network to obtain an updated detection network; The first sample set is used to train the initial detection network to obtain the detection network, including: The initial detection network is used to detect positive samples in the first sample set to obtain a first feature difference sequence corresponding to the positive sample; the first feature difference sequence is determined based on the difference between a reconstructed feature sequence and a multi-scale feature sequence, the reconstructed feature sequence is determined based on the reconstruction of the multi-scale feature sequence and a preset query word sequence, and the multi-scale feature is obtained based on the initial detection network; Determining the training loss corresponding to the positive sample according to the first feature difference sequence and a normal loss function; Based on the training loss, the initial detection network is trained and adjusted until the final loss obtained is less than a preset loss threshold, thereby obtaining the detection network; The normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is brought closer to the positive sample.

2. The method according to claim 1, characterized in that The step of obtaining a second sample set based on the detected normal image set and abnormal image set comprises: Determining an image to be verified from the detected normal image set and the abnormal image set; The image to be verified is verified to obtain the second sample set.

3. The method according to claim 2, characterized in that The images to be verified include: at least one abnormal image and at least one normal image; and verifying the images to be verified to obtain the second sample set includes: Verifying the at least one abnormal image and the at least one normal image respectively to obtain respective verification results; The abnormal image whose verification result indicates that it is correct is used as the negative sample, and the normal image whose verification result indicates that it is correct is used as the positive sample; Correctly labeling the abnormal image whose verification result indicates an error and the normal image whose verification result indicates an error to obtain the positive sample and the negative sample; A set of the negative samples and the positive samples is determined as the second sample set.

4. The method according to any one of claims 1 to 3, characterized in that: After the initial detection network is trained using the first sample set to obtain the detection network, the method further includes: Using the detection network to perform anomaly detection on each image to be detected, and obtaining an anomaly score at each pixel position of each image to be detected; Based on the anomaly score, draw an anomaly detection image corresponding to each image to be detected; According to the abnormal detection image, the normal image set and the abnormal image set in a plurality of images to be detected are obtained.

5. The method according to claim 4, characterized in that The step of obtaining the normal image set and the abnormal image set in a plurality of images to be detected according to the abnormal detection image comprises: In the case where each image to be detected is an abnormal image, each image to be detected is used as an abnormal image in the abnormal detection image representation corresponding to the image to be detected, and after traversing the multiple images to be detected, the abnormal image set including at least one abnormal image is obtained; In the abnormal detection image representation corresponding to each image to be detected, when each image to be detected is a normal image, each image to be detected is taken as a normal image, and after traversing the multiple images to be detected, the normal image set containing at least one normal image is obtained.

6. The method according to claim 2, characterized in that The step of determining the image to be verified from the detected normal image set and the abnormal image set comprises: Determine, from each abnormal image in the abnormal image set, a first maximum abnormality score that is the largest among the abnormality scores at each pixel position; Determine, from each normal image in the normal image set, a second maximum anomaly score which is the largest among the anomaly scores at each pixel position; When the first maximum anomaly score belongs to a first preset value range of a preset anomaly threshold, determining that the abnormal image corresponding to the first maximum anomaly score belongs to the image to be verified; When the second maximum anomaly score belongs to the second preset value range of the preset anomaly threshold, it is determined that the normal image corresponding to the second maximum anomaly score belongs to the image to be verified.

7. The method according to claim 2, characterized in that The step of determining the image to be verified from the detected normal image set and the abnormal image set comprises: All images in the normal image set and the abnormal image set are determined as the images to be verified.

8. The method according to claim 1, characterized in that The adopting the second sample set to perform updating training on the detection network to obtain an updated detection network includes: When the detection network is trained using positive samples in the second sample set, the detection network is used to obtain a first feature difference sequence corresponding to the positive sample, and a first training loss corresponding to the positive sample is determined based on the first feature difference sequence and a normal loss function; When the detection network is trained using negative samples in the second sample set, the detection network is used to obtain a second feature difference sequence corresponding to the negative sample, and a second training loss corresponding to the negative sample is determined according to the second feature difference sequence, a preset true value of the negative sample, and an abnormal loss function; Based on the first training loss and the second training loss, respectively, the detection network is trained and adjusted until the final loss obtained is less than a preset loss threshold, thereby obtaining the updated detection network; Among them, the normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is brought closer to the positive sample; the abnormal loss function represents that the reconstructed feature sequence corresponding to the abnormal part in the negative sample is pushed away from the abnormal part, and the reconstructed feature sequence corresponding to the normal part in the negative sample is brought closer to the normal part.

9. The method according to any one of claims 5 to 8, characterized in that: The updated detection network includes: an updated feature extraction network and an updated reconstruction network; After the detection network is updated and trained using the second sample set to obtain an updated detection network, the method further includes: Using the updated feature extraction network, performing feature processing of different scales on the target image to obtain a multi-scale feature sequence; Using the updated reconstruction network, reconstructing the multi-scale feature sequence and the preset query word sequence to obtain a reconstructed feature sequence; Determining a feature difference at each pixel position of the target image according to the reconstructed feature sequence and the multi-scale feature sequence; Based on the feature difference, an abnormality score at each pixel position is determined, and based on the abnormality score, an abnormality detection image corresponding to the target image is drawn.

10. An abnormality detection device, characterized in that: include: A first training unit is used to train an initial detection network using a first sample set to obtain a detection network; The first sample set is a positive sample set; An acquisition unit is used to obtain a second sample set based on the detected normal image set and abnormal image set during the abnormality detection of the plurality of images to be detected by the detection network, wherein the second sample set is an incremental sample set including positive samples and negative samples; wherein the negative sample is an abnormal image in which an abnormality exists; A second training unit is used to use the second sample set to update the detection network to obtain an updated detection network; The first training unit is further used to use the initial detection network to detect the positive samples in the first sample set to obtain a first feature difference sequence corresponding to the positive sample; the first feature difference sequence is determined based on the difference between the reconstructed feature sequence and the multi-scale feature sequence, the reconstructed feature sequence is determined based on the reconstruction of the multi-scale feature sequence and a preset query word sequence, and the multi-scale feature is obtained based on the initial detection network; according to the first feature difference sequence and the normal loss function, the training loss corresponding to the positive sample is determined; based on the training loss, the initial detection network is trained and adjusted until the final loss obtained is less than a preset loss threshold, and the detection network is obtained; wherein the normal loss function represents that the reconstructed feature sequence corresponding to the positive sample is close to the positive sample.

11. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: A computer program is stored, which is used to implement the method described in any one of claims 1 to 9 when executed by a processor.

Images