Cross-domain small sample anomaly detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of anomaly detection, and provides a cross-domain small sample anomaly detection method and device, electronic equipment and a storage medium, and the method comprises the steps: extracting text features of image content knowledge corresponding to a normal image; determining a test image reconstruction result of the test image based on a matching difference between the image sub-block features of the test image and the image sub-block features of the normal image, and determining a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result; determining an intermediate semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; and determining a high-level semantic anomaly detection result based on the text features and the image overall features of the test image, and determining a target anomaly detection result based on the low-level, middle-level and high-level semantic anomaly detection results. According to the method, a large number of samples do not need to be used for training, so that the anomaly detection training cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of anomaly detection technology, and in particular to a cross-domain small sample anomaly detection method, device, electronic device and storage medium. Background Art

[0002] Anomaly detection aims to use computer vision technology to detect whether there are visible anomalies in the sample to be tested by taking pictures or scanning images. It is a practical task involving multiple fields, including defect detection in industrial production, logical anomaly detection in product assembly, lesion detection in the medical field, abnormal vehicle detection in traffic scenes, etc. Existing anomaly detection technologies usually design dedicated detection frameworks and methods for each field, such as the image sub-block feature matching method commonly used in industrial defect detection, the image component segmentation-based method commonly used in logical anomaly detection, and the image reconstruction-based method commonly used in medical and traffic anomaly detection.

[0003] However, the existing anomaly detection technology has the following main shortcomings: (1) Existing anomaly detection technology requires the design of dedicated detection methods for different data characteristics and anomaly types in each field. Anomaly detection methods designed for one field are difficult to apply to data in another field. (2) In the same field, existing anomaly detection methods usually require the training of dedicated models for each type of object to be detected. The trained model can only be used to detect anomalies for this type of object, and the generalization performance is poor. (3) Existing anomaly detection technology requires the use of a large number of samples for training, which requires a large amount of training data and a cumbersome training process.

[0004] In summary, existing anomaly detection technologies are highly specialized and customized, making them difficult to expand and generalize, which limits the standardized research on anomaly detection technologies. Summary of the invention

[0005] The present invention provides a cross-domain small sample anomaly detection method, device, electronic device and storage medium to solve the defects of the prior art that the existing anomaly detection technology is highly specialized and customized, difficult to expand and generalize, and limits the standardization research of anomaly detection technology.

[0006] The present invention provides a cross-domain small sample anomaly detection method, comprising the following steps: Acquire image content knowledge of a normal image and a test image respectively, and extract text features of the image content knowledge corresponding to the normal image; determine a test image reconstruction result of the test image based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, and determine a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result; Determining a mid-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; Based on the text features and the overall image features of the test image, a high-level semantic anomaly detection result in the test image is determined, and based on the low-level semantic anomaly detection result, the intermediate semantic anomaly detection result and the high-level semantic anomaly detection result, a target anomaly detection result of the test image is determined.

[0007] According to a cross-domain small sample anomaly detection method provided by the present invention, the test image reconstruction result of the test image is determined based on the matching difference between the first image sub-block feature of the test image and the second image sub-block feature of the normal image, including: sorting the matching differences between the first image sub-block feature of the test image and the second image sub-block feature of the normal image from small to large to obtain a target matching difference with the smallest matching difference; Determine a first target image sub-block feature in the normal image corresponding to the target matching difference, and a second target image sub-block feature in the test image corresponding to the first target image sub-block feature; The second target image sub-block feature in the test image is replaced with the first target image sub-block feature to obtain the test image reconstruction result.

[0008] According to a cross-domain small sample anomaly detection method provided by the present invention, the low-level semantic anomaly detection result in the test image is determined based on the reconstruction difference corresponding to the reconstruction result of the test image, including: Determine a reconstruction difference between a feature of each image sub-block in the reconstruction result of the test image and a feature of each image sub-block in the test image; When the reconstruction difference is greater than a first threshold, it is determined that the image sub-block feature in the test image is a low-level semantic anomaly.

[0009] According to a cross-domain small sample anomaly detection method provided by the present invention, the first sub-component feature is determined based on the image content knowledge of the test image and the first image sub-block feature of the test image; The second sub-component feature is determined based on image content knowledge of the normal image and a second image sub-block feature of the normal image.

[0010] According to a cross-domain small sample anomaly detection method provided by the present invention, the image content knowledge includes all objects in the image, detection frames of all the objects, and segmentation masks of the objects in the detection frames; The step of determining the first image sub-block feature comprises: Multiplying the segmentation mask in the image content knowledge of the test image by the test image pixel by pixel to obtain a first target sub-block; Using a feature extractor to extract features from the first target sub-block to obtain features of the first image sub-block; The step of determining the second image sub-block feature comprises: Multiplying the segmentation mask in the image content knowledge of the normal image by the normal image pixel by pixel to obtain a second target sub-block; The feature extractor is used to perform feature extraction on the second target sub-block to obtain features of the second image sub-block.

[0011] According to a cross-domain small sample anomaly detection method provided by the present invention, the intermediate semantic anomaly detection result in the test image is determined based on the first sub-component feature of the test image and the second sub-component feature of the normal image, including: determining a subcomponent anomaly score between the first subcomponent feature and the second subcomponent feature; When the sub-component anomaly score is greater than a second threshold, it is determined that the sub-component feature in the test image is a mid-level semantic anomaly.

[0012] According to a cross-domain small sample anomaly detection method provided by the present invention, the high-level semantic anomaly detection result in the test image is determined based on the text features and the overall image features of the test image, including: Determining an overall anomaly score between the text features of the normal image and the overall image features of the test image; When the overall anomaly score is greater than a third threshold, it is determined that the overall image feature of the test image is a high-level semantic anomaly.

[0013] The present invention also provides a cross-domain small sample anomaly detection device, comprising the following units: An acquisition unit, used to respectively acquire image content knowledge of a normal image and a test image, and extract text features of the image content knowledge corresponding to the normal image; a first determining unit, configured to determine a test image reconstruction result of the test image based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, and to determine a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result; A second determining unit, configured to determine a mid-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; a third determination unit, configured to determine a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and to determine a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate semantic anomaly detection result and the high-level semantic anomaly detection result.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the cross-domain small sample anomaly detection method as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the cross-domain small sample anomaly detection methods described above.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the cross-domain small sample anomaly detection methods described above.

[0017] The cross-domain small sample anomaly detection method, device, electronic device and storage medium provided by the present invention (1) realize a cross-domain anomaly detection method by classifying different types of anomalies in different fields into three categories: low-level semantic anomalies, intermediate semantic anomalies and high-level semantic anomalies, and detecting these three types of anomalies; (2) perform detailed content analysis on normal images and test images, thereby alleviating the knowledge gap between different fields, providing image content knowledge for subsequent anomaly detection, and improving cross-domain anomaly detection performance; (3) use a combination of feature comparison-based and reconstruction-based detection methods, only a small number of normal samples need to be provided during the test to realize anomaly detection of the test image, without the need for training on proprietary data, thus solving the problem of large training requirements of existing methods; (4) by comprehensively comparing features at different semantic levels, as well as text features and overall image features, the present method can detect different types of anomalies that may exist in the test image, thereby realizing cross-domain small sample anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the cross-domain small sample anomaly detection method provided by the present invention.

[0020] Figure 2 It is a schematic diagram of a process for determining a low-level semantic anomaly detection result in a test image provided by the present invention.

[0021] Figure 3 This is the second flow chart of the cross-domain small sample anomaly detection method provided by the present invention.

[0022] Figure 4 This is the third flow chart of the cross-domain small sample anomaly detection method provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the cross-domain small sample anomaly detection device provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0027] Figure 1 This is one of the flow charts of the cross-domain small sample anomaly detection method provided by the present invention. Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0028] Step 110, respectively obtaining image content knowledge of a normal image and a test image, and extracting text features of the image content knowledge corresponding to the normal image.

[0029] Specifically, the image content knowledge of the normal image and the image content knowledge of the test image can be obtained. For example, the visual basic model can be used to perform content recognition, target detection, and semantic segmentation on the image contents in the normal image and the test image, respectively, to obtain the image content knowledge in the normal image and the image content knowledge in the test image.

[0030] Here, normal images refer to expected or standard images, which are usually used to train models to help the model learn features under normal circumstances. Test images are images used to verify model performance, which are usually not involved in training and may contain normal or abnormal situations.

[0031] The image content knowledge includes all objects in the image, detection frames of all objects, and segmentation masks of objects in the detection frames. Here, all objects are, for example, wooden boards, nails, fruits, etc., which are not specifically limited in the embodiment of the present invention.

[0032] Then, the text features of the image content knowledge corresponding to the normal image are extracted. Specifically, a text description corresponding to the image content knowledge of the normal image can be written, and the text features of the text description corresponding to the normal image can be extracted using a text encoder.

[0033] Among them, the text encoder can be a BERT (Bidirectional Encoder Representations from Transformers) model, or an RNN (Recurrent Neural Network), or a CNN (Convolutional Neural Network), etc., which is not specifically limited in the embodiment of the present invention.

[0034] Step 120, determining a test image reconstruction result of the test image based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, and determining a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result.

[0035] Specifically, Figure 2 is a flow chart of determining the low-level semantic anomaly detection result in the test image provided by the present invention, such as Figure 2 As shown, the test image reconstruction result of the test image can be determined based on the matching difference between the first image sub-block feature of the test image and the second image sub-block feature of the normal image. For example, bipartite graph matching can be performed on the first image sub-block feature of the test image and the second image sub-block feature of the normal image.

[0036] Here, an image encoder may be used to extract features from a normal image to obtain features of a second image sub-block, and an image encoder may be used to extract features from a test image to obtain features of a first image sub-block.

[0037] The matching difference may be calculated using cosine distance or Manhattan distance, etc., which is not specifically limited in the embodiment of the present invention.

[0038] Then, based on the reconstruction difference corresponding to the test image reconstruction result, the low-level semantic anomaly detection result in the test image is determined. The reconstruction difference corresponding to the test image reconstruction result may be determined based on the difference between the test image and the test image reconstruction result.

[0039] The difference between the test image and the test image reconstruction result may be determined using cosine distance or Manhattan distance, which is not specifically limited in the embodiment of the present invention.

[0040] Here, the low-level semantic anomaly detection result is used to reflect the low-level semantic anomaly of the test image. The low-level semantic anomaly refers to an extremely local anomaly occurring in a certain image sub-block, such as scratches and damage on a wooden board. The embodiment of the present invention does not make any specific limitation on this.

[0041] Step 130 : determining a mid-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image.

[0042] Specifically, the intermediate semantic anomaly detection result in the test image may be determined based on the first sub-component feature of the test image and the second sub-component feature of the normal image.

[0043] Here, the intermediate semantic anomaly detection result is used to reflect the intermediate semantic anomaly of the test image. The intermediate semantic anomaly refers to the anomaly that occurs at a sub-component level in the target. For example, a mechanical component consists of two screws and two nuts, but it is detected that a certain component has only one screw or one nut, then the intermediate semantic anomaly detection result is an intermediate semantic anomaly.

[0044] Step 140, determining a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and determining a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate semantic anomaly detection result and the high-level semantic anomaly detection result.

[0045] Specifically, the high-level semantic anomaly detection result in the test image can be determined based on the text features and the overall image features of the test image. Here, the high-level semantic anomaly detection result is used to reflect the high-level semantic anomaly of the test image. The high-level semantic anomaly refers to the anomaly of the test image as a whole, such as an iron plate appearing during the detection of a wooden board.

[0046] The high-level semantic anomaly detection result may be determined based on an overall anomaly score between text features and overall image features of the test image. The overall anomaly score may be calculated using cosine distance or Manhattan distance, etc., which is not specifically limited in the embodiments of the present invention.

[0047] After obtaining the low-level semantic anomaly detection results, the intermediate semantic anomaly detection results and the high-level semantic anomaly detection results, the target anomaly detection score can be determined based on the low-level semantic anomaly detection results, the intermediate semantic anomaly detection results and the high-level semantic anomaly detection results, and the target anomaly detection result of the test image can be determined based on the target anomaly detection score.

[0048] Here, the formula for the target anomaly detection score is as follows: in, represents the target anomaly detection score, represents the low-level semantic anomaly detection results, represents the mid-level semantic anomaly detection result, represents the high-level semantic anomaly detection results, is an adjustable weighting hyperparameter.

[0049] It should be noted that, when the target anomaly detection score is greater than a preset threshold, the target anomaly detection result is determined to be a test image anomaly.

[0050] The method provided by the embodiment of the present invention (1) realizes a cross-domain anomaly detection method by classifying different types of anomalies in different fields into three categories: low-level semantic anomalies, intermediate semantic anomalies, and high-level semantic anomalies, and detecting these three types of anomalies; (2) performs detailed content analysis on normal images and test images, which can alleviate the knowledge gap between different fields, provide image content knowledge for subsequent anomaly detection, and improve cross-domain anomaly detection performance; (3) comprehensively uses feature comparison-based and reconstruction-based detection methods, only a small number of normal samples need to be provided during the test to realize anomaly detection of the test image, without the need for training on proprietary data, thus solving the problem of large training requirements of existing methods; (4) by comprehensively comparing features at different semantic levels, as well as text features and overall image features, the method can detect different types of anomalies that may exist in the test image, and realize cross-domain small sample anomaly detection.

[0051] Based on the above embodiment, the step 120 of determining the test image reconstruction result of the test image based on the matching difference between the first image sub-block feature of the test image and the second image sub-block feature of the normal image includes: Step 121, sorting the matching differences between the first image sub-block feature of the test image and the second image sub-block feature of the normal image from small to large, and obtaining a target matching difference with the smallest matching difference; Step 122, determining a first target image sub-block feature in the normal image corresponding to the target matching difference, and a second target image sub-block feature in the test image corresponding to the first target image sub-block feature; Step 123: Replace the second target image sub-block feature in the test image with the first target image sub-block feature to obtain the test image reconstruction result.

[0052] Specifically, the matching differences between the first image sub-block feature of the test image and the second image sub-block feature of the normal image are sorted from small to large to obtain a target matching difference with the smallest matching difference.

[0053] Here, the matching difference can be calculated using the cosine distance, and the formula for the matching difference is as follows: in, Indicates matching differences, represents the first image sub-block feature, Represents the second image sub-block feature.

[0054] Then, a first target image sub-block feature in the normal image corresponding to the target matching difference and a second target image sub-block feature in the test image corresponding to the first target image sub-block feature are determined.

[0055] Finally, the second target image sub-block feature in the test image is replaced with the first target image sub-block feature to obtain the test image reconstruction result, thereby realizing the reconstruction of the test image. The reconstruction formula is as follows: in, represents the test image reconstruction result, represents the second target image sub-block feature, Represents the first target image sub-block feature.

[0056] Based on the above embodiment, determining the low-level semantic anomaly detection result in the test image based on the reconstruction difference corresponding to the reconstruction result of the test image in step 120 includes: Step 210, determining a reconstruction difference between features of each image sub-block in the reconstruction result of the test image and features of each image sub-block in the test image; Step 220: When the reconstruction difference is greater than a first threshold, determine that the image sub-block feature in the test image is a low-level semantic anomaly.

[0057] Specifically, the reconstruction difference between the features of each image sub-block in the test image reconstruction result and the features of each image sub-block in the test image is determined, and whether each sub-block of the test image is abnormal is judged according to the size of the reconstruction difference.

[0058] The formula for reconstructing the difference is as follows: in, represents the reconstruction difference, represents the test image reconstruction result, Represents a test image.

[0059] When the reconstruction difference is greater than the first threshold, it is determined that the image sub-block feature in the test image is a low-level semantic anomaly.

[0060] Based on the above embodiment, the first sub-component feature is determined based on the image content knowledge of the test image and the first image sub-block feature of the test image; The second sub-component feature is determined based on image content knowledge of the normal image and a second image sub-block feature of the normal image.

[0061] Specifically, the first sub-component feature is determined based on the image content knowledge of the test image and the first image sub-block feature of the test image, and the formula is as follows: , in, represents the first subcomponent feature, Each subcomponent i The number of image sub-blocks contained in represents the first image sub-block feature of the test image, Represents the image content knowledge of the test image.

[0062] Here, the second sub-component feature is determined based on the image content knowledge of the normal image and the second image sub-block feature of the normal image, and the formula is as follows: in, represents the second subcomponent feature, Each subcomponent i The number of image sub-blocks contained in The second image sub-block feature representing the normal image, Represents the image content knowledge of the test image.

[0063] Based on the above embodiment, the image content knowledge includes all objects in the image, detection frames of all the objects, and segmentation masks of the objects in the detection frames; The step of determining the first image sub-block feature comprises: Multiplying the segmentation mask in the image content knowledge of the test image by the test image pixel by pixel to obtain a first target sub-block; Using a feature extractor to extract features from the first target sub-block to obtain features of the first image sub-block; The step of determining the second image sub-block feature comprises: Multiplying the segmentation mask in the image content knowledge of the normal image by the normal image pixel by pixel to obtain a second target sub-block; The feature extractor is used to perform feature extraction on the second target sub-block to obtain features of the second image sub-block.

[0064] Specifically, the image content knowledge includes all objects in the image, the detection boxes of all objects, and the segmentation masks of the objects in the detection boxes.

[0065] Specifically, we can use the visual target recognition basic model to identify all targets contained in the image, then use the visual target detection basic model to detect the identified targets and obtain the detection boxes of all targets. Finally, we use the visual semantic segmentation basic model to segment all targets in all detection boxes and obtain the segmentation mask of each target.

[0066] The visual target recognition basic model may be a Recognize Anything model, the visual target detection basic model may be a Grounding DINO model, the visual semantic segmentation basic model may be a Segment Anything model, etc., which is not specifically limited in the embodiment of the present invention.

[0067] Here, the step of determining the first image sub-block feature includes: The segmentation mask in the image content knowledge of the test image is multiplied pixel by pixel with the test image to obtain the first target sub-block. Here, the segmentation mask represents the target area in the form of a binary image, where the area with a mask value of 1 represents the target object and the area with a mask value of 0 represents the background.

[0068] Then, a feature extractor is used to extract features of the first target sub-block to obtain features of the first image sub-block. The feature extractor may be a CNN model or other feature extractors.

[0069] Here, the step of determining the second image sub-block feature includes: The segmentation mask in the image content knowledge of the normal image is multiplied pixel by pixel with the normal image to obtain the second target sub-block, and then the feature extractor is used to extract features from the second target sub-block to obtain the second image sub-block feature. The first image sub-block feature and the second image sub-block feature refer to the features extracted from the local area (i.e., sub-block) of the image, which are used to describe the visual information of the area.

[0070] Based on the above embodiment, step 130 includes: Step 131, determining a sub-component anomaly score between the first sub-component feature and the second sub-component feature; Step 132: When the sub-component anomaly score is greater than a second threshold, determine that the sub-component feature in the test image is a mid-level semantic anomaly.

[0071] Specifically, the sub-component anomaly score between the first sub-component feature and the second sub-component feature is determined. For example, the first sub-component feature and the second sub-component feature may be compared to each other to determine whether each sub-component of the test image contains an anomaly. The formula is as follows: in, represents the subcomponent anomaly score, represents the first subcomponent feature, represents the second subcomponent feature, Represents the cosine distance between the first subcomponent feature and the second subcomponent feature.

[0072] When the sub-component anomaly score is greater than the second threshold, it is determined that the sub-component feature in the test image is a mid-level semantic anomaly.

[0073] Based on the above embodiment, the step 140 of determining a high-level semantic anomaly detection result in the test image based on the text feature and the overall image feature of the test image includes: Step 141, determining an overall anomaly score between the text features of the normal image and the overall image features of the test image; Step 142: When the overall anomaly score is greater than a third threshold, determine that the overall image feature of the test image is a high-level semantic anomaly.

[0074] Specifically, the overall anomaly score between the text features of the normal image and the overall image features of the test image can be determined to determine whether the overall content in the test image has changed, and based on this, the high-level semantic anomalies of the overall level of the test image can be detected. The formula is as follows: in, represents the overall anomaly score, Represents the overall characteristics of the image, Represents the text features of normal images, Represents the cosine distance between the overall image features and the text features.

[0075] Based on any of the above embodiments, Figure 3 This is the second flow chart of the cross-domain small sample anomaly detection method provided by the present invention. Figure 4 This is the third flow chart of the cross-domain small sample anomaly detection method provided by the present invention. Figure 3 , Figure 4 As shown, the method includes: In the first step, the visual base model is used to perform content recognition, object detection and semantic segmentation on normal images and test images respectively to obtain image content knowledge, and the text encoder is used to extract the text features of the image content knowledge corresponding to the normal images.

[0076] In the second step, an image encoder is used to extract feature vectors of the test image and the normal image, including the feature vector of the entire image and the feature vector of each image sub-block.

[0077] The third step is to perform bipartite graph matching on the first image sub-block features of the test image and the second image sub-block features of the normal image, and sort the matching differences from small to large to obtain the target matching difference with the smallest matching difference, and then determine the first target image sub-block features in the normal image corresponding to the target matching difference, and the second target image sub-block features in the test image corresponding to the first target image sub-block features. Finally, replace the second target image sub-block features in the test image with the first target image sub-block features to obtain the test image reconstruction result.

[0078] The fourth step is to determine the reconstruction difference between each image sub-block feature in the test image reconstruction result and each image sub-block feature in the test image. When the reconstruction difference is greater than the first threshold, the image sub-block feature in the test image is determined to be a low-level semantic anomaly.

[0079] In the fifth step, the first sub-component feature of the test image and the second sub-component feature of the normal image are compared to the nearest neighbor to determine the intermediate semantic anomaly detection result in the test image. The first sub-component feature is determined based on the image content knowledge of the test image and the first image sub-block feature of the test image, and the second sub-component feature is determined based on the image content knowledge of the normal image and the second image sub-block feature of the normal image.

[0080] In the sixth step, based on the text features and the overall image features of the test image, the high-level semantic anomaly detection results in the test image are determined, and based on the low-level semantic anomaly detection results, the intermediate semantic anomaly detection results and the high-level semantic anomaly detection results, the target anomaly detection results are determined.

[0081] The cross-domain small sample anomaly detection device provided by the present invention is described below. The cross-domain small sample anomaly detection device described below and the cross-domain small sample anomaly detection method described above can be referenced to each other.

[0082] Based on any of the above embodiments, the present invention provides a cross-domain small sample anomaly detection device. Figure 5 is a schematic diagram of the structure of the cross-domain small sample anomaly detection device provided by the present invention, such as Figure 5 As shown, the device comprises: An acquisition unit 510 is used to respectively acquire image content knowledge of a normal image and a test image, and extract text features of the image content knowledge corresponding to the normal image; A first determining unit 520 is configured to determine a test image reconstruction result of the test image based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, and determine a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result; A second determining unit 530, configured to determine a mid-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; The third determination unit 540 is used to determine a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and to determine a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate semantic anomaly detection result and the high-level semantic anomaly detection result.

[0083] The device provided by the embodiment of the present invention (1) realizes a cross-domain anomaly detection method by classifying different types of anomalies in different fields into three categories: low-level semantic anomalies, intermediate semantic anomalies, and high-level semantic anomalies, and detecting these three types of anomalies; (2) performs detailed content analysis on normal images and test images, which can alleviate the knowledge gap between different fields, provide image content knowledge for subsequent anomaly detection, and improve cross-domain anomaly detection performance; (3) comprehensively uses feature comparison-based and reconstruction-based detection methods, only a small number of normal samples need to be provided during the test to realize anomaly detection of the test image, without the need for training on proprietary data, thus solving the problem of large training requirements of existing methods; (4) by comprehensively comparing features at different semantic levels, as well as text features and overall image features, the method can detect different types of anomalies that may exist in the test image, and realize cross-domain small sample anomaly detection.

[0084] Based on any of the foregoing embodiments, the first determining unit 520 is specifically configured to: sorting the matching differences between the first image sub-block feature of the test image and the second image sub-block feature of the normal image from small to large to obtain a target matching difference with the smallest matching difference; Determine a first target image sub-block feature in the normal image corresponding to the target matching difference, and a second target image sub-block feature in the test image corresponding to the first target image sub-block feature; The second target image sub-block feature in the test image is replaced with the first target image sub-block feature to obtain the test image reconstruction result.

[0085] Based on any of the foregoing embodiments, the first determining unit 520 is specifically configured to: Determine a reconstruction difference between a feature of each image sub-block in the reconstruction result of the test image and a feature of each image sub-block in the test image; When the reconstruction difference is greater than a first threshold, it is determined that the image sub-block feature in the test image is a low-level semantic anomaly.

[0086] Based on any of the above embodiments, the first sub-component feature is determined based on image content knowledge of the test image and a first image sub-block feature of the test image; The second sub-component feature is determined based on image content knowledge of the normal image and a second image sub-block feature of the normal image.

[0087] Based on any of the above embodiments, the image content knowledge includes all objects in the image, detection frames of all the objects, and segmentation masks of the objects in the detection frames; The invention also includes a first image sub-block feature determination unit, wherein the first image sub-block feature determination unit is specifically configured to: Multiplying the segmentation mask in the image content knowledge of the test image by the test image pixel by pixel to obtain a first target sub-block; Using a feature extractor to extract features from the first target sub-block to obtain features of the first image sub-block; The invention also includes a second image sub-block feature determination unit, wherein the second image sub-block feature determination unit is specifically configured to: Multiplying the segmentation mask in the image content knowledge of the normal image by the normal image pixel by pixel to obtain a second target sub-block; The feature extractor is used to perform feature extraction on the second target sub-block to obtain features of the second image sub-block.

[0088] Based on any of the foregoing embodiments, the second determining unit 530 is specifically configured to: determining a subcomponent anomaly score between the first subcomponent feature and the second subcomponent feature; When the sub-component anomaly score is greater than a second threshold, it is determined that the sub-component feature in the test image is a mid-level semantic anomaly.

[0089] Based on any of the foregoing embodiments, the third determining unit 540 is specifically configured to: Determining an overall anomaly score between the text features of the normal image and the overall image features of the test image; When the overall anomaly score is greater than a third threshold, it is determined that the overall image feature of the test image is a high-level semantic anomaly.

[0090] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a cross-domain small sample anomaly detection method, which includes: respectively obtaining image content knowledge of a normal image and a test image, and extracting text features of the image content knowledge corresponding to the normal image; determining a test image reconstruction result of the test image based on the matching difference between the first image sub-block feature of the test image and the second image sub-block feature of the normal image, and determining a low-level semantic anomaly detection result in the test image based on the reconstruction difference corresponding to the test image reconstruction result; determining an intermediate-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; determining a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and determining a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate-level semantic anomaly detection result and the high-level semantic anomaly detection result.

[0091] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-domain small sample anomaly detection method provided by the above methods, the method including: respectively obtaining image content knowledge of a normal image and a test image, and extracting text features of the image content knowledge corresponding to the normal image; determining a test image reconstruction result of the test image based on the matching difference between the first image sub-block feature of the test image and the second image sub-block feature of the normal image, and determining a low-level semantic anomaly detection result in the test image based on the reconstruction difference corresponding to the test image reconstruction result; determining an intermediate-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; determining a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and determining a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate-level semantic anomaly detection result and the high-level semantic anomaly detection result.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the cross-domain small sample anomaly detection method provided by the above-mentioned methods, the method comprising: respectively acquiring image content knowledge of a normal image and a test image, and extracting text features of the image content knowledge corresponding to the normal image; determining a test image reconstruction result of the test image based on the matching difference between the first image sub-block feature of the test image and the second image sub-block feature of the normal image, and determining a low-level semantic anomaly detection result in the test image based on the reconstruction difference corresponding to the test image reconstruction result; determining an intermediate-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; determining a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and determining a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate-level semantic anomaly detection result and the high-level semantic anomaly detection result.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-domain small sample anomaly detection method, characterized in that: include: Acquire image content knowledge of a normal image and a test image respectively, and extract text features of the image content knowledge corresponding to the normal image; determining a test image reconstruction result of the test image based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, and determining a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result; Determining a mid-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; Based on the text features and the overall image features of the test image, a high-level semantic anomaly detection result in the test image is determined, and based on the low-level semantic anomaly detection result, the intermediate semantic anomaly detection result and the high-level semantic anomaly detection result, a target anomaly detection result of the test image is determined.

2. The cross-domain small sample anomaly detection method according to claim 1, characterized in that: The determining, based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, a test image reconstruction result of the test image comprises: sorting the matching differences between the first image sub-block feature of the test image and the second image sub-block feature of the normal image from small to large to obtain a target matching difference with the smallest matching difference; Determine a first target image sub-block feature in the normal image corresponding to the target matching difference, and a second target image sub-block feature in the test image corresponding to the first target image sub-block feature; The second target image sub-block feature in the test image is replaced with the first target image sub-block feature to obtain the test image reconstruction result.

3. The cross-domain small sample anomaly detection method according to claim 1, characterized in that: The determining, based on the reconstruction difference corresponding to the reconstruction result of the test image, a low-level semantic anomaly detection result in the test image, comprises: Determine a reconstruction difference between a feature of each image sub-block in the reconstruction result of the test image and a feature of each image sub-block in the test image; When the reconstruction difference is greater than a first threshold, it is determined that the image sub-block feature in the test image is a low-level semantic anomaly.

4. The cross-domain small sample anomaly detection method according to any one of claims 1 to 3, characterized in that: The first sub-component feature is determined based on image content knowledge of the test image and a first image sub-block feature of the test image; The second sub-component feature is determined based on image content knowledge of the normal image and a second image sub-block feature of the normal image.

5. The cross-domain small sample anomaly detection method according to any one of claims 1 to 3, characterized in that: The image content knowledge includes all objects in the image, detection frames of all the objects, and segmentation masks of the objects in the detection frames; The step of determining the first image sub-block feature comprises: Multiplying the segmentation mask in the image content knowledge of the test image by the test image pixel by pixel to obtain a first target sub-block; Using a feature extractor to extract features from the first target sub-block to obtain features of the first image sub-block; The step of determining the second image sub-block feature comprises: Multiplying the segmentation mask in the image content knowledge of the normal image by the normal image pixel by pixel to obtain a second target sub-block; The feature extractor is used to perform feature extraction on the second target sub-block to obtain features of the second image sub-block.

6. The cross-domain small sample anomaly detection method according to any one of claims 1 to 3, characterized in that: The determining, based on the first sub-component feature of the test image and the second sub-component feature of the normal image, a mid-level semantic anomaly detection result in the test image comprises: determining a subcomponent anomaly score between the first subcomponent feature and the second subcomponent feature; When the sub-component anomaly score is greater than a second threshold, it is determined that the sub-component feature in the test image is a mid-level semantic anomaly.

7. The cross-domain small sample anomaly detection method according to any one of claims 1 to 3, characterized in that: The step of determining a high-level semantic anomaly detection result in the test image based on the text feature and the overall image feature of the test image includes: Determining an overall anomaly score between the text features of the normal image and the overall image features of the test image; When the overall anomaly score is greater than a third threshold, it is determined that the overall image feature of the test image is a high-level semantic anomaly.

8. A cross-domain small sample anomaly detection device, characterized in that: include: An acquisition unit, used to respectively acquire image content knowledge of a normal image and a test image, and extract text features of the image content knowledge corresponding to the normal image; a first determining unit, configured to determine a test image reconstruction result of the test image based on a matching difference between a first image sub-block feature of the test image and a second image sub-block feature of the normal image, and to determine a low-level semantic anomaly detection result in the test image based on a reconstruction difference corresponding to the test image reconstruction result; A second determining unit, configured to determine a mid-level semantic anomaly detection result in the test image based on the first sub-component feature of the test image and the second sub-component feature of the normal image; a third determination unit, configured to determine a high-level semantic anomaly detection result in the test image based on the text features and the overall image features of the test image, and to determine a target anomaly detection result of the test image based on the low-level semantic anomaly detection result, the intermediate semantic anomaly detection result and the high-level semantic anomaly detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the cross-domain small sample anomaly detection method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-domain small sample anomaly detection method as described in any one of claims 1 to 7 is implemented.

Images