Abnormality detection method and device, electronic terminal and computer readable storage medium

Abstract

The invention provides an anomaly detection method and device, an electronic terminal and a computer readable storage medium, and the method comprises the steps: carrying out the feature extraction of an obtained to-be-detected image, and obtaining a plurality of depth feature maps of the to-be-detected image; determining an abnormal thermodynamic diagram of the to-be-detected image based on the plurality of depth feature maps corresponding to the to-be-detected image; the depth feature maps are different in size; and obtaining an anomaly detection result of the to-be-detected image according to the anomaly thermodynamic diagram of the to-be-detected image. And determining an abnormal thermodynamic diagram of the to-be-detected image through the plurality of depth feature maps corresponding to the to-be-detected image, so as to determine an abnormal detection result of the to-be-detected image according to the abnormal thermodynamic diagram. Compared with an existing anomaly detection method, the method can achieve the anomaly detection of the to-be-detected image without a training model, saves the training cost and time, improves the detection efficiency, and improves the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an abnormality detection method, device, electronic terminal and computer-readable storage medium. Background Art

[0002] Anomaly detection is of great significance to industrial production. Previous anomaly detection methods are highly dependent on data, often requiring the collection and labeling of large amounts of data, which is time-consuming and laborious. In addition, for different scenarios and products, the model needs to be retrained to adapt to new scenarios. The existing solution's dependence on data makes it very difficult to quickly deploy anomaly detection models. Summary of the invention

[0003] The main technical problem solved by the present invention is to provide an anomaly detection method, device, electronic terminal and computer-readable storage medium, which can improve detection efficiency and detection accuracy.

[0004] In a first aspect, the present application provides an anomaly detection method, the anomaly detection method comprising: Performing feature extraction on the acquired image to be detected to obtain multiple depth feature maps of the image to be detected; Based on multiple depth feature maps corresponding to the image to be detected, an abnormal heat map of the image to be detected is determined; the sizes of the depth feature maps are different; the abnormal heat map of the image to be detected is obtained according to the abnormal heat maps of each depth feature map; the abnormal heat map of the depth feature map is determined according to the similarity between the depth feature map and other depth feature maps; According to the abnormal heat map of the image to be detected, the abnormality detection result of the image to be detected is obtained.

[0005] In the technical solution of the embodiment of the present application, the abnormal heat map of the image to be detected is determined by multiple depth feature maps corresponding to the image to be detected, so as to determine the abnormal detection result of the image to be detected according to the abnormal heat map. Compared with the existing abnormality detection method, the abnormality detection of the image to be detected can be realized without training the model, which saves training cost and time, and improves both detection efficiency and detection accuracy.

[0006] In some embodiments, determining an abnormal heat map of the image to be detected based on a plurality of depth feature maps corresponding to the image to be detected includes: Traversing multiple depth feature maps, and taking each depth feature map in turn as a candidate depth feature map; Determine an abnormal heat map of the candidate depth feature map based on a similarity between the candidate depth feature map and each of the multiple depth feature maps except the candidate depth feature map; The abnormal heat maps of each depth feature map are summed and averaged to obtain the abnormal heat map of the image to be detected.

[0007] In the technical solution of the embodiment of the present application, the abnormal heat map of the depth feature map is determined according to the similarity between the depth feature map and the depth feature maps of other layers, thereby improving the accuracy of the abnormal heat map of the deep feature map; then, the abnormal heat map of the image to be detected is obtained according to the abnormal heat map of the depth feature maps of each layer, thereby improving the accuracy of the abnormal heat map of the image to be detected.

[0008] In some embodiments, determining an abnormal heat map of a candidate depth feature map based on a similarity between the candidate depth feature map and each of the multiple depth feature maps except the candidate depth feature map includes: Determine the similarity between the candidate depth feature map and each corresponding pixel in each of the multiple depth feature maps except the candidate depth feature map; The similarities between each pixel in the candidate depth feature map and the corresponding pixel in other depth feature maps are added and averaged to obtain the normal index of each pixel in the candidate depth feature map; Based on the normal index of the pixel in the candidate depth feature map, determine the abnormal index of the pixel; the sum of the normal index and the abnormal index of the pixel is 1; Based on the anomaly index of each pixel in the candidate depth feature map, an anomaly heat map of the candidate depth feature map is determined.

[0009] In the technical solution of the embodiment of the present application, the degree of difference between the pixel points in the depth feature map and the pixel points of other depth feature maps is determined based on the similarity between the pixel points at the corresponding positions of the depth feature map and the depth feature maps of other layers; then, the abnormal heat map of each depth feature map is determined based on the degree of difference between the pixel points in the depth feature map, thereby improving the accuracy of the abnormal heat map corresponding to the depth feature map.

[0010] In some embodiments, feature extraction is performed on the acquired image to be detected to obtain multiple depth feature maps of the image to be detected, including: Performing image enhancement processing on the image to be detected to obtain an enhanced image of the image to be detected; the image enhancement processing includes geometric transformation and / or color transformation; Feature extraction is performed on the enhanced image of the image to be detected to obtain multiple depth feature maps of the image to be detected.

[0011] In the technical solution of the embodiment of the present application, the quality of the image to be detected is improved by performing image enhancement processing on the image to be detected, so as to improve the detection accuracy.

[0012] In some embodiments, obtaining an abnormality detection result of the image to be detected according to the abnormality heat map of the image to be detected includes: Binarize the abnormal heat map of the image to be detected to obtain a mask image of the image to be detected; Identify the mask image of the image to be detected and determine the abnormal area of ​​the image to be detected; Based on the abnormal area of ​​the image to be detected, a minimum bounding rectangle of the abnormal area is generated; Based on the minimum bounding rectangle of the abnormal area and the image to be detected, an abnormality detection result of the image to be detected is obtained.

[0013] In the technical solution of the embodiment of the present application, the abnormal heat map of the image to be detected is binarized to facilitate determining the abnormal area of ​​the image to be detected, and a detection frame of the abnormal area in the image to be detected is generated, thereby reducing the error due to manual annotation of the abnormal detection frame and improving the objectivity and accuracy of the abnormal area detection frame; the detection frame of the abnormal area can be obtained without training the model, saving the model training cost and time, and improving both the detection efficiency and the detection accuracy.

[0014] In some embodiments, before the step of binarizing the abnormal heat map of the image to be detected to obtain the mask image of the image to be detected, the method further includes: The abnormal thermal map of the image to be detected is preprocessed, and the preprocessing includes median filtering, edge feature suppression and / or contrast enhancement.

[0015] In the technical solution of the embodiment of the present application, the abnormal heat map of the image to be detected is preprocessed to improve the quality of the abnormal heat map, so as to facilitate the subsequent improvement of the edge accuracy of the abnormal area.

[0016] In some embodiments, based on the minimum bounding rectangle of the abnormal area and the image to be detected, obtaining an abnormality detection result of the image to be detected includes: SAM is used to obtain the abnormal segmentation result of the image to be detected based on the minimum bounding rectangle of the abnormal area and the image to be detected.

[0017] In the technical solution of the embodiment of the present application, the abnormal area in the image to be detected is segmented by SAM, so as to improve the segmentation accuracy of the abnormal area in the image to be detected and ensure that the method has good generalization performance.

[0018] In a second aspect, the present application provides an anomaly detection device, the anomaly detection device comprising: A feature extraction module is used to extract features from the acquired image to be detected to obtain multiple depth feature maps of the image to be detected; An analysis module is used to determine an abnormal heat map of the image to be detected based on multiple depth feature maps corresponding to the image to be detected; the sizes of the depth feature maps are different; the abnormal heat map of the image to be detected is obtained according to the abnormal heat maps of each depth feature map; the abnormal heat map of the depth feature map is determined according to the similarity between the depth feature map and other depth feature maps; The detection module is used to obtain an abnormality detection result of the image to be detected based on the abnormality heat map of the image to be detected.

[0019] In a third aspect, the present application provides an electronic terminal, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor is used to execute program data to implement the steps in the anomaly detection method of the first aspect as described above.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the anomaly detection method in the first aspect described above are implemented.

[0021] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a flowchart of the anomaly detection method provided by this application; Figure 2 yes Figure 1 A flowchart of a specific embodiment of step S2 in the provided anomaly detection method; Figure 3 yes Figure 1 A flowchart of a specific embodiment of step S3 in the provided anomaly detection method; Figure 4 It is a schematic block diagram of a specific embodiment of SAM in the anomaly detection method provided in the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of an abnormality detection device provided by the present application; Figure 6 It is a schematic diagram of the framework of an embodiment of an electronic terminal provided by the present application; Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium provided in the present application. DETAILED DESCRIPTION

[0025] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0027] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0028] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0030] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0031] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0032] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0033] The Segment Anything Model (SAM) uses the Transformer architecture, which has been shown to be very effective in image recognition tasks. Specifically, SAM uses a visual transformer-based image encoder to extract image features and merge user interactions with a prompt encoder, and then uses a mask decoder to generate segmentation results and confidence scores based on image embeddings, prompt embeddings, and output tags. Among them, SAM includes an image encoder, a prompt encoder, and a mask decoder.

[0034] SAM has powerful automatic segmentation capabilities and wide versatility, and can accept prompt information such as clicks, boxes, and text for further precise segmentation.

[0035] Currently, when detecting whether a product has defects, different text prompt information needs to be set for different products in order to detect the location information of the defect edge. However, some defects are difficult to describe using text. Due to various difficulties, the anomaly detection method that uses text and does not require training is still difficult to use in actual industrial production.

[0036] When obtaining prompt information, it is necessary to rely on the target detection model to detect the abnormal area prompt box in the image, and then use SAM to obtain the abnormal area in the image based on the abnormal area prompt box and the image. For example, the target detection model can be Grounding DINO. However, different target detection models are required for defects of different products, and target detection models suitable for different scenarios and products need to be trained with different training data. Since the target detection model needs to rely on training data for training, the rapid deployment of the target detection model becomes very difficult. In addition, different target detection models require different types of training data for training, and different training data need to be labeled, which is time-consuming and labor-intensive.

[0037] Therefore, this application proposes an anomaly detection method that does not require training of a target detection model. The method is applicable to different scenarios and products, can autonomously generate prompt information, and can obtain anomaly detection results of the image to be detected through SAM without human intervention.

[0038] The anomaly detection method provided in the embodiment of the present application can be implemented by the server or the terminal alone, or by the server and the terminal in collaboration. In some embodiments, the terminal or the server can implement the anomaly detection method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0039] See also Figure 1 , Figure 1 It is a flowchart of the anomaly detection method provided by this application.

[0040] The following introduces an anomaly detection method provided by this embodiment. This specification provides method operation steps such as the embodiment or flowchart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. The method can be applied to terminals such as computers, tablets, smart phones, smart wearable devices, and vehicle-mounted devices. Of course, it can also be applied to servers according to actual needs.

[0041] S1: extracting features from the acquired image to be detected to obtain multiple depth feature maps of the image to be detected.

[0042] S2: Determine an abnormal heat map of the image to be detected based on multiple depth feature maps corresponding to the image to be detected; the sizes of each depth feature map are different; the abnormal heat map of the image to be detected is obtained based on the abnormal heat maps of each depth feature map; the abnormal heat map of the depth feature map is determined based on the similarity between the depth feature map and other depth feature maps.

[0043] S3: Obtain an anomaly detection result of the image to be detected according to the anomaly heat map of the image to be detected.

[0044] In the technical solution of the embodiment of the present application, the abnormal heat map of the image to be detected is determined by multiple depth feature maps corresponding to the image to be detected, so as to determine the abnormal detection result of the image to be detected according to the abnormal heat map of the image to be detected. Compared with the existing abnormality detection method, this technical solution can realize abnormality detection of the image to be detected without training the model, saving training cost and time, and improving both detection efficiency and detection accuracy.

[0045] In some embodiments, the step of performing feature extraction on the acquired image to be detected in step S1 to obtain multiple depth feature maps of the image to be detected specifically includes the following implementation methods.

[0046] Specifically, the image to be detected can be understood as an image that needs to be detected for abnormality, such as image information of a product that needs to be detected. The number of images to be detected can be one or more, and the size, dimensions, color, etc. of the image to be detected can also be set according to actual needs, and the embodiments of this specification do not specifically limit it. The scheme provided in the embodiments of this specification can be used to perform abnormality detection on the image to be detected, and the abnormal area in the image to be detected can be determined.

[0047] In a specific embodiment, a feature extraction module is used to extract features from the image to be detected to obtain multiple depth feature maps of different depths. The depth (i.e., the number of channels) of each depth feature map extracted by the feature extraction module gradually increases and the size gradually decreases, and each layer extracts more meaningful information from the depth feature map of the previous layer, while reducing the spatial dimension to increase computational efficiency. Among them, the sizes of multiple depth feature maps are different. Among them, the feature extraction module can be a residual network (Residual Networks, ResNets), a visual geometry group (Visual Geometry Group, VGG) network, a convolutional neural network (Convolutional Neural Networks, CNN), a feature pyramid network (FeaturePyramid Networks, FPN), a fully convolutional network (Fully Convolutional Networks, FCN), etc., which can be set according to actual conditions. The feature extraction module can be pre-trained.

[0048] In a specific embodiment, an image enhancement process is performed on the image to be detected to obtain an enhanced image of the image to be detected; the image enhancement process includes geometric transformation and / or color transformation; and feature extraction is performed on the enhanced image of the image to be detected to obtain multiple depth feature maps of the image to be detected. Among them, the geometric transformation includes rotation, translation, scaling, flipping, distortion, etc.; the color transformation includes brightness adjustment, contrast adjustment, saturation adjustment, hue adjustment, etc.

[0049] In the technical solution of the embodiment of the present application, the quality of the image to be detected is improved by performing image enhancement processing on the image to be detected, so as to improve the detection accuracy.

[0050] For example, an image to be inspected is acquired by performing image acquisition on the product to be inspected. The resolution of the image to be inspected may be 1024*1024 and the format may be RGB. The product may be a part. The image to be inspected is enhanced to obtain an enhanced image corresponding to the image to be inspected. The enhanced image is subjected to feature extraction by a feature extraction module to obtain multiple depth feature maps output by each feature extraction layer. The size of the depth feature map may be H×W×c, where H and W represent the height and width of the depth feature map, and c represents the number of feature channels of the feature extraction module. For example, the size of the depth feature may be 64×64×512.

[0051] See also Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific embodiment of step S2 in the provided anomaly detection method is provided.

[0052] In some embodiments, the step of determining the abnormal heat map of the image to be detected based on multiple depth feature maps corresponding to the image to be detected in step S2 specifically includes the following implementation methods.

[0053] S21: traverse multiple depth feature maps, and take each depth feature map as a candidate depth feature map in turn.

[0054] S22: Determine an abnormal heat map of the candidate depth feature map based on the similarity between the candidate depth feature map and other depth feature maps among the multiple depth feature maps except the candidate depth feature map.

[0055] Specifically, determine the similarity between the candidate depth feature map and each corresponding pixel in each of the multiple depth feature maps except the candidate depth feature map; add and average the similarities between each pixel in the candidate depth feature map and the corresponding pixel in each of the other depth feature maps to obtain a normal index of each pixel in the candidate depth feature map; based on the normal index of the pixel in the candidate depth feature map, determine the abnormal index of the pixel; the sum of the normal index and the abnormal index of the pixel is 1; based on the abnormal index of each pixel in the candidate depth feature map, determine the abnormal heat map of the candidate depth feature map.

[0056] In a specific embodiment, the abnormal heat map of the candidate depth feature map is determined by the following formula.

[0057] (Formula 1) Where: D i is the abnormal heat map of the deep feature map of the i-th layer, which is a c-dimensional vector; N(z i ) is the feature set most similar to the deep feature map, and the dimension of the deep feature map is H×W; |N(z i )| is the number of features in the feature set; <z i , z j > is the cosine similarity between the deep feature map of the jth layer and the deep feature map of the i-th layer.

[0058] In the technical solution of the embodiment of the present application, the degree of difference between the pixel points in the depth feature map and the pixel points of other depth feature maps is determined based on the similarity between the pixel points at the corresponding positions of the depth feature map and the depth feature maps of other layers; then, the abnormal heat map of each depth feature map is determined based on the degree of difference between the pixel points in the depth feature map, thereby improving the accuracy of the abnormal heat map corresponding to the depth feature map.

[0059] Through the above steps, the abnormal heat map of each depth feature map can be obtained.

[0060] S23: The abnormal heat maps of each depth feature map are summed and averaged to obtain the abnormal heat map of the image to be detected.

[0061] Specifically, the abnormal heat map corresponding to each depth feature map of the image to be detected is added and averaged to obtain the abnormal heat map of the image to be detected. The abnormal heat map represents the abnormality score of each pixel in the image to be detected. The value range of the abnormality score is [0, 1]. The higher the abnormality score, the higher the possibility that the pixel is abnormal.

[0062] In the technical solution of the embodiment of the present application, the abnormal heat map of the depth feature map is determined according to the similarity between the depth feature map and the depth feature maps of other layers, thereby improving the accuracy of the abnormal heat map of the deep feature map; then, the abnormal heat map of the image to be detected is obtained according to the abnormal heat map of the depth feature maps of each layer, thereby improving the accuracy of the abnormal heat map of the image to be detected.

[0063] In a specific embodiment, the abnormal thermal map of the image to be detected may be preprocessed, and the preprocessing includes median filtering, edge feature suppression and / or contrast enhancement.

[0064] Among them, median filtering is a nonlinear signal processing method used to remove noise in image processing. The basic idea of ​​median filtering is to replace the value of each pixel in the image with the median of the pixel values ​​in its domain. This method can effectively smooth the image while keeping the edges clear. Edge Feature Suppression is an image processing technology that is mainly used to reduce edge features in an image to achieve the purpose of smoothing the image or removing noise. Contrast Enhancement is an image processing technology used to improve the visual quality and interpretability of an image. The purpose of contrast enhancement is to make different areas or features in the image more obvious, thereby increasing the contrast of the image and making the image look clearer and more layered.

[0065] In the technical solution of the embodiment of the present application, the abnormal heat map of the image to be detected is preprocessed to improve the quality of the abnormal heat map, so as to facilitate the subsequent improvement of the edge accuracy of the abnormal area.

[0066] See also Figure 3 , Figure 3 yes Figure 1 A flowchart of a specific embodiment of step S3 in the provided anomaly detection method is provided.

[0067] In some embodiments, the step of obtaining an abnormality detection result of the image to be detected according to the abnormality heat map of the image to be detected in step S3 specifically includes the following implementation methods.

[0068] S31: Binarize the abnormal heat map of the image to be detected to obtain a mask image of the image to be detected.

[0069] Specifically, the anomaly score of each pixel in the anomaly heat map of the image to be detected is compared with the preset value. When the anomaly score of the pixel is higher than the preset value, the pixel corresponding to the anomaly score is assigned a first identifier; when the anomaly score of the pixel is not higher than the preset value, the pixel corresponding to the anomaly score is assigned a second identifier. After binarization, the mask image of the image to be detected is obtained. Among them, the preset value can be 80% or 75%, etc., which is set according to the actual scene.

[0070] S32: Identify the mask image of the image to be detected and determine the abnormal area of ​​the image to be detected.

[0071] Specifically, the mask image of the image to be detected is identified to obtain edge pixels corresponding to the area composed of pixels assigned with the first identifier, and the area surrounded by the edge pixels is used as the abnormal area of ​​the image to be detected. For example, the abnormal area of ​​the image to be detected is a circular area with a diameter of 20 pixels, and the abnormality score of each pixel in the circular area is much higher than the abnormality score of pixels in other areas.

[0072] Through this step, the precise position of the abnormal area in the image to be detected can be obtained.

[0073] S33: Based on the abnormal area of ​​the image to be detected, generate a minimum circumscribed rectangle of the abnormal area.

[0074] Specifically, an outer edge frame is generated outside the abnormal area of ​​the image to be detected, and the outer edge frame serves as prompt information in subsequent steps. The outer edge frame includes the abnormal area. In a specific embodiment, a minimum circumscribed rectangular frame is generated in the abnormal area of ​​the image to be detected, and then the abnormal position in the image to be detected is accurately framed as prompt information. The prompt information is used to indicate the area of ​​interest in the image to be detected.

[0075] Through the above steps, the abnormal position in the image to be detected can be generated autonomously, and the minimum circumscribed rectangular frame of the abnormal area in the image to be detected can be obtained as prompt information, so as to realize automatic generation of prompt information and improve detection efficiency. Specifically, it is possible to automatically identify abnormal areas in the image without any text or manual prompts. This method not only improves the detection efficiency, but also improves the subjectivity and errors caused by manual labeling, and improves the objectivity and accuracy of the detection results.

[0076] No pre-trained target detection model is required to detect abnormal areas, which reduces costs and also reduces the errors caused by manual selection of abnormal areas.

[0077] S34: Based on the minimum bounding rectangle of the abnormal area and the image to be detected, an abnormality detection result of the image to be detected is obtained.

[0078] Specifically, SAM is used based on the minimum bounding rectangle of the abnormal area and the image to be detected to obtain the abnormal segmentation result of the image to be detected.

[0079] See also Figure 4 , Figure 4 It is a schematic block diagram of a specific embodiment of SAM in the anomaly detection method provided in this application.

[0080] In a specific embodiment, SAM includes an image encoder, a hint encoder, and a mask decoder. The image to be detected is sliced ​​by the image encoder to generate multiple image blocks, and the multiple image blocks are embedded as pictures. Image embedding is a vector representation of image data, which captures the key information in the image to facilitate subsequent processing by the model. The minimum bounding rectangle of the abnormal area is encoded into a hint embedding by the hint encoder, wherein the hint embedding contains the position information of the minimum bounding rectangle. Weights are assigned to image blocks at each position based on the mask image of the image to be detected. Specifically, the mask image is feature extracted by the convolution layer, and the weights of each image block are summed with the image blocks. The picture embedding of the image to be detected is fused with the hint embedding by the mask decoder to guide the model to focus on specific areas; and the mask decoder is used to decode the fused feature map and hint information to generate a predicted mask map and a predicted mask confidence.

[0081] In the technical solution of the embodiment of the present application, the abnormal area of ​​the image to be detected is segmented by SAM, so as to improve the segmentation accuracy of the abnormal area in the image to be detected and ensure that the method has good generalization performance.

[0082] In the technical solution of the embodiment of the present application, the abnormal heat map of the image to be detected is binarized to facilitate determining the abnormal area of ​​the image to be detected, and a detection frame of the abnormal area in the image to be detected is generated, thereby reducing the error caused by manual annotation of the abnormal detection frame and improving the objectivity and accuracy of the detection frame of the abnormal area; the detection frame of the abnormal area can be obtained without training the model, saving the model training cost and time, and improving both the detection efficiency and the detection accuracy.

[0083] The anomaly detection method provided by this application reduces data dependence and simplifies model training. Specifically, traditional anomaly detection methods often require a large amount of labeled data for model training, while this application significantly reduces the dependence on data through unsupervised learning, simplifies the model training process, and thus accelerates the deployment and application of the model.

[0084] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of an abnormality detection device provided in the present application.

[0085] This embodiment provides an abnormality detection device 60 , which includes a feature extraction module 61 , an analysis module 62 and a detection module 63 .

[0086] The feature extraction module 61 is used to extract features from the acquired image to be detected, and obtain multiple depth feature maps of the image to be detected.

[0087] The analysis module 62 is used to determine the abnormal heat map of the image to be detected based on multiple depth feature maps corresponding to the image to be detected; the sizes of each depth feature map are different; the abnormal heat map of the image to be detected is obtained based on the abnormal heat map of each depth feature map; the abnormal heat map of the depth feature map is determined based on the similarity between the depth feature map and other depth feature maps.

[0088] The detection module 63 is used to obtain an abnormality detection result of the image to be detected according to the abnormality thermal map of the image to be detected.

[0089] The anomaly detection device provided in this embodiment determines the anomaly heat map of the image to be detected through multiple depth feature maps corresponding to the image to be detected, so as to determine the anomaly detection result of the image to be detected according to the anomaly heat map of the image to be detected. Compared with the existing anomaly detection method, anomaly detection of the image to be detected can be achieved without training a model, saving training costs and time, and improving both detection efficiency and detection accuracy.

[0090] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of an electronic terminal provided by the present application.

[0091] This embodiment provides an electronic terminal 80, which includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is used to execute a computer program stored in the memory 81 to implement the steps of any of the above-mentioned abnormality detection method embodiments. In a specific implementation scenario, the electronic terminal 80 may include but is not limited to: a microcomputer, a server, and in addition, the electronic terminal 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0092] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of any of the above-mentioned abnormality detection method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0093] See also Figure 7 , Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium provided in the present application.

[0094] This embodiment provides a computer-readable storage medium 90 , which stores program instructions 901 that can be executed by a processor, and the program instructions 901 are used to implement the steps of any of the above-mentioned abnormality detection method embodiments.

[0095] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0096] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0098] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0100] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An anomaly detection method, characterized in that: The anomaly detection method comprises: Performing feature extraction on the acquired image to be detected to obtain multiple depth feature maps of the image to be detected; Based on the multiple depth feature maps corresponding to the image to be detected, determining an abnormal heat map of the image to be detected; the sizes of the depth feature maps are different; the abnormal heat map of the image to be detected is obtained according to the abnormal heat maps of the depth feature maps; the abnormal heat map of the depth feature map is determined according to the similarity between the depth feature map and other depth feature maps; According to the abnormal heat map of the image to be detected, an abnormality detection result of the image to be detected is obtained.

2. The anomaly detection method according to claim 1, characterized in that: The step of determining an abnormal heat map of the image to be detected based on the multiple depth feature maps corresponding to the image to be detected includes: Traversing the multiple depth feature maps, and taking each of the depth feature maps in turn as a candidate depth feature map; Determine an abnormal heat map of the candidate depth feature map based on a similarity between the candidate depth feature map and each of the multiple depth feature maps except the candidate depth feature map; The abnormal heat maps of the depth feature maps are added and averaged to obtain the abnormal heat map of the image to be detected.

3. The anomaly detection method according to claim 2, characterized in that: The determining, based on the similarity between the candidate depth feature map and each of the multiple depth feature maps except the candidate depth feature map, an abnormal heat map of the candidate depth feature map comprises: Determine the similarity between the candidate depth feature map and corresponding pixels in each of the depth feature maps other than the candidate depth feature map; The similarities between each pixel in the candidate depth feature map and the corresponding pixel in other depth feature maps are added and averaged to obtain a normal index of each pixel in the candidate depth feature map; Based on the normal index of the pixel in the candidate depth feature map, determining the abnormal index of the pixel; the sum of the normal index of the pixel and the abnormal index is 1; Based on the abnormal index of each pixel in the candidate depth feature map, an abnormal heat map of the candidate depth feature map is determined.

4. The abnormality detection method according to any one of claims 1 to 3, characterized in that: The step of extracting features from the acquired image to be detected to obtain a plurality of depth feature maps of the image to be detected includes: Performing image enhancement processing on the image to be detected to obtain an enhanced image of the image to be detected; the image enhancement processing includes geometric transformation and / or color transformation; Feature extraction is performed on the enhanced image of the image to be detected to obtain multiple depth feature maps of the image to be detected.

5. The abnormality detection method according to any one of claims 1 to 3, characterized in that: The step of obtaining an abnormality detection result of the image to be detected according to the abnormality heat map of the image to be detected includes: Binarizing the abnormal thermal map of the image to be detected to obtain a mask image of the image to be detected; Identifying the mask image of the image to be detected to determine the abnormal area of ​​the image to be detected; Based on the abnormal area of ​​the image to be detected, generating a minimum circumscribed rectangle of the abnormal area; Based on the minimum circumscribed rectangle of the abnormal area and the image to be detected, an abnormality detection result of the image to be detected is obtained.

6. The anomaly detection method according to claim 5, characterized in that: Before the step of binarizing the abnormal heat map of the image to be detected to obtain the mask image of the image to be detected, the method further includes: The abnormal thermal map of the image to be detected is preprocessed, and the preprocessing includes median filtering, edge feature suppression and / or contrast enhancement.

7. The abnormality detection method according to claim 5, characterized in that: The obtaining of an abnormality detection result of the image to be detected based on the minimum circumscribed rectangle of the abnormal area and the image to be detected includes: SAM is used to obtain an abnormality segmentation result of the image to be detected based on the minimum circumscribed rectangle of the abnormal area and the image to be detected.

8. An abnormality detection device, characterized in that: The abnormality detection device comprises: A feature extraction module is used to extract features from the acquired image to be detected to obtain multiple depth feature maps of the image to be detected; An analysis module, configured to determine an abnormal heat map of the image to be detected based on the multiple depth feature maps corresponding to the image to be detected; the sizes of the depth feature maps are different; the abnormal heat map of the image to be detected is obtained based on the abnormal heat maps of the depth feature maps; the abnormal heat map of the depth feature map is determined based on the similarity between the depth feature map and other depth feature maps; The detection module is used to obtain an abnormality detection result of the image to be detected based on the abnormality heat map of the image to be detected.

9. An electronic terminal, characterized in that: The electronic terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to execute program data to implement the steps in the abnormality detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the anomaly detection method according to any one of claims 1 to 7 are implemented.

Images