Artificial intelligence-based anomaly detection method, device, equipment and storage medium

Abstract

Embodiments of the present application provide an abnormality detection method and device based on artificial intelligence, equipment and storage medium, which are related to the fields of artificial intelligence, image detection, cloud technology and the like. The method comprises: extracting a plurality of levels of first feature maps of a to-be-detected image by using a trained abnormality detection model, and reconstructing the first feature maps of at least part of the levels based on an intermediate feature map to obtain a reconstructed feature map corresponding to the first feature map of each level in the at least part of the levels, wherein the intermediate feature map at least includes the first feature map of the last level; obtaining an abnormality map corresponding to each level in M levels based on the first feature maps of the M levels and the reconstructed feature maps corresponding to the first feature maps of the M levels, and identifying an abnormality detection result of the to-be-detected image based on the abnormality maps corresponding to the M levels. The method provided in the embodiments of the present application can better meet the actual application requirements.

Description

Technical Field

[0001] This application belongs to the field of computer technology and may relate to fields such as artificial intelligence, image detection, and cloud technology. Specifically, this application relates to an anomaly detection method, apparatus, device, and storage medium based on artificial intelligence. Background Technology

[0002] With the rapid development of science and technology and the improvement of people's living standards, the demand for anomaly detection (AD) applications has been increasing in recent years. Especially in many practical industrial scenarios, anomaly detection is indispensable. For example, in industrial product manufacturing applications, it is necessary to detect whether the generated products are abnormal (whether the products are qualified), and in some scenarios, it is necessary to detect whether the current scene environment is abnormal.

[0003] To meet the ever-increasing application demands, various anomaly detection technologies have been proposed. In recent years, with the rapid development of artificial intelligence, AI technology has also been increasingly applied to the field of anomaly detection. While numerous AI-based anomaly detection solutions have been proposed in related research, these solutions are currently difficult to widely implement in industrial scenarios, and their detection effectiveness still needs improvement. Therefore, current anomaly detection technologies still require further refinement. Summary of the Invention

[0004] The purpose of this application is to provide an artificial intelligence-based anomaly detection method, device, electronic device, and storage medium that can better meet the needs of practical applications. To achieve this purpose, the technical solutions provided by this application are as follows:

[0005] On the one hand, embodiments of this application provide an anomaly detection method based on artificial intelligence, the method comprising:

[0006] Acquire the image to be detected; use a trained anomaly detection model to extract features from the image to be detected, obtain first feature maps of multiple levels, and reconstruct the first feature maps of at least some levels of the multiple levels based on intermediate feature maps, to obtain reconstructed feature maps corresponding to the first feature maps of each of the at least some levels, wherein the intermediate feature maps belong to the first feature maps of the multiple levels and include at least the first feature map of the last level of the multiple levels;

[0007] Based on the first feature maps of M levels and the reconstructed feature maps corresponding to the first feature maps of the M levels, anomaly maps corresponding to each of the M levels are obtained, where M is a positive integer. An anomaly map corresponding to a level is used to characterize the pixel difference between the first feature map and the reconstructed feature map of that level. The M levels belong to the at least some levels.

[0008] The anomaly detection results of the image to be detected are identified based on the anomaly maps corresponding to the M levels;

[0009] The trained anomaly detection model is obtained by training the anomaly detection model to be trained based on non-abnormal sample images and training loss. The training loss is used to characterize the difference between the M-level sample feature maps obtained by feature extraction of the sample images and the reconstructed sample feature maps corresponding to the M-level sample feature maps.

[0010] Optionally, the resolution of the first feature map and the reconstructed feature map at each level is the same.

[0011] Optionally, the training loss is determined based on the M levels of anomaly maps corresponding to the sample image. Alternatively, the training loss is determined based on the sum of pixel values ​​in the M levels of anomaly maps corresponding to the sample image.

[0012] On the other hand, embodiments of this application provide an anomaly detection device based on artificial intelligence, the device comprising:

[0013] The image acquisition module is used to acquire the image to be detected.

[0014] The feature extraction and reconstruction module is used to extract features from the image to be detected using a trained anomaly detection model to obtain first feature maps at multiple levels, and to reconstruct the first feature maps at least some of the multiple levels based on intermediate feature maps to obtain reconstructed feature maps corresponding to the first feature maps at each of the at least some levels, wherein the intermediate feature maps belong to the first feature maps of the multiple levels and include at least the first feature map of the last level among the multiple levels, and the at least some levels do not include the last level;

[0015] An anomaly determination module is used to obtain an anomaly map corresponding to each of the M levels based on the first feature maps of the M levels and the reconstructed feature maps corresponding to the first feature maps of the M levels, and to identify the anomaly detection result of the image to be detected based on the anomaly maps corresponding to the M levels, wherein M is a positive integer, an anomaly map corresponding to a level is used to characterize the pixel difference between the first feature map and the reconstructed feature map of the level, and the M levels belong to the at least some levels;

[0016] The trained anomaly detection model is obtained by training the anomaly detection model to be trained based on non-abnormal sample images and training loss. The training loss is used to characterize the difference between the M-level sample feature maps obtained by feature extraction of the sample images and the reconstructed sample feature maps corresponding to the M-level sample feature maps.

[0017] Optionally, the resolution of the first feature map and the reconstructed feature map at each level is the same.

[0018] Optionally, the anomaly determination module can be used to: fuse the anomaly maps corresponding to the M levels to obtain the final anomaly map, and identify the anomaly detection result of the image to be detected based on the final anomaly map.

[0019] Optionally, the anomaly determination module can be used to: obtain the pixel value of the corresponding position in the final anomaly map by adding the pixel values ​​of the same position in the anomaly maps corresponding to the M levels.

[0020] Optionally, the anomaly determination module can identify the anomaly detection result of the image to be detected by performing at least one of the following:

[0021] The final abnormal image is divided into regions and pixel values ​​are fused. The fused result of each pixel value in each region is used as the abnormal score of that region. Based on the abnormal scores of each region, it is identified whether there is an abnormality in the image to be detected. The size of each region is a set size.

[0022] The pixel value of each position in the final abnormal image is used as the abnormal score for that position. If there is a position in the final abnormal image with an abnormal score not less than the second threshold, the image region corresponding to each position in the final abnormal image with an abnormal score not less than the second threshold in the image to be detected is determined as an abnormal region.

[0023] Optionally, both the first feature map and the reconstructed feature map of each level have multiple channels, and the number of channels in the first feature map and the number of channels in the reconstructed feature map of a level are the same; for each of the M levels, the anomaly map corresponding to that level is obtained in the following way:

[0024] For each identical position in the first feature map and the reconstructed feature map of this level, the pixel values ​​of that position in multiple channels of the first feature map are combined to obtain the original feature vector corresponding to that position, and the pixel values ​​of that position in multiple channels of the reconstructed feature map are combined to obtain the reconstructed feature vector corresponding to that position.

[0025] For each identical position, the degree of difference between the original feature vector and the reconstructed feature vector corresponding to that position is determined, and the degree of difference is used as the pixel value of that position in the anomaly map corresponding to that level.

[0026] Optionally, the image acquisition module is configured to: determine the anomaly detection task type, wherein the task type is a first type or a second type, the first type is used to detect whether there is an anomaly in the image, and the second type is used to detect abnormal regions in the image; when the task type is the first type, acquire the image to be detected with a resolution of a first set resolution; when the task type is the second type, acquire the image to be detected with a resolution of a second set resolution, wherein the second set resolution is greater than the first set resolution.

[0027] Optionally, the at least partial levels include all levels except the last level among the plurality of levels, and the M levels include all levels except the first level among the plurality of levels.

[0028] Optionally, the trained anomaly detection model includes an encoder and a decoder, wherein the encoder includes multiple cascaded encoding modules and the decoder includes multiple cascaded decoding modules.

[0029] The first feature maps of the multiple layers, and the reconstructed feature maps corresponding to the first feature maps of each of the at least some layers, are obtained in the following manner:

[0030] The image to be detected is input into the encoder, and features are extracted sequentially by each encoding module of the encoder to obtain a first feature map of a level corresponding to each encoding module. The input of the next encoding module is the first feature map output by the previous encoding module.

[0031] Based on the intermediate feature map, the first feature map of each level in the at least some levels is reconstructed by each decoding module in the decoder to obtain the reconstructed feature map corresponding to the first feature map of each level in the at least some levels.

[0032] Optionally, the anomaly detection model further includes a feature fusion unit connecting the encoder and the decoder, wherein the reconstructed feature map corresponding to the first feature map of each of at least some of the aforementioned layers is obtained in the following manner:

[0033] The feature fusion processor performs feature fusion on the first feature maps of each level in the intermediate feature map to obtain a fused feature map.

[0034] The fused feature map is input into the decoder, and the reconstructed feature map corresponding to the first feature map of each of the at least some layers is obtained by each of the decoding modules.

[0035] Optionally, the neural network structures of each encoding module and each decoding module are identical.

[0036] Optionally, the encoder and the decoder are based on a neural network structure of the Naive Vision Transformer (ViT) network, each encoding module includes at least one cascaded feature extraction layer, and each decoding module includes at least one cascaded feature reconstruction layer. The neural network structure satisfies at least one of the following:

[0037] Each feature extraction layer and each feature reconstruction layer correspond to a Transformer encoding module in the ViT network, and the Transformer encoding module does not include a normalization layer;

[0038] The inputs and outputs of the Transformer encoding module do not include classification token features;

[0039] The feature fusion unit includes at least one linear layer.

[0040] Optionally, the above feature extraction and reconstruction module can be used for:

[0041] The image to be detected is divided into multiple image blocks according to the target size;

[0042] For each image block, an image feature vector is obtained based on the pixel information of the image block, and a position feature vector is obtained based on the position of the image block in the image to be detected. An initial feature vector of the image block is obtained by fusing the image feature vector and the position feature vector.

[0043] The initial feature vectors of the multiple image patches are used as input to the encoder, and features are extracted sequentially by each encoding module in the encoder.

[0044] Optionally, the above fused feature map is obtained in the following way:

[0045] The feature fusion processor concatenates the first feature maps of each level in the intermediate feature map along the channel dimension, and performs feature transformation on the concatenated feature map to obtain a fused feature map with the number of channels equal to the number of output channels of the encoding module.

[0046] Optionally, the feature extraction and reconstruction module is further used for:

[0047] Determine the anomaly detection task type, which is either a first type or a second type. The first type is used to detect whether there is an anomaly in the image, and the second type is used to detect abnormal regions in the image.

[0048] If the task type is a first type, the first size is determined as the target size; if the task type is a second type, the second size is determined as the target size, and the second size is smaller than the first size.

[0049] Optionally, the trained anomaly detection model is trained in the following way:

[0050] Obtain a first training set, which includes multiple labeled training images, where the label of a training image represents the true category of that training image.

[0051] The first neural network model is trained based on the first training set until the first training termination condition is met. The first neural network model includes an encoder of the anomaly detection model to be trained and a classifier cascaded with the encoder. The input and output of the first neural network model are the training image and the predicted category of the training image, respectively. The training loss of the first neural network model represents the difference between the true category and the predicted category of each training image.

[0052] Obtain a second training set, which includes multiple non-abnormal sample images (i.e., sample images without abnormalities), wherein the multiple sample images include images of sample objects of multiple categories;

[0053] The encoder that meets the first training termination condition is taken as the trained encoder. The model parameters of the trained encoder are fixed. The part of the anomaly detection model to be trained, excluding the encoder, is trained based on the second training set until the second training termination condition is met, and the trained anomaly detection model is obtained.

[0054] Optionally, based on the second training set, the part of the anomaly detection model to be trained, excluding the encoder, is used to train the model to obtain a trained anomaly detection model, including:

[0055] The training operation is repeated on the anomaly detection model to be trained based on the second training set until the second training termination condition is met, thereby obtaining a trained anomaly detection model. The training operation includes:

[0056] Each of the sample images is input into the anomaly detection model to be trained, resulting in M-level sample feature maps and reconstructed sample feature maps corresponding to each sample image. Based on the differences between the M-level sample feature maps and the reconstructed sample feature maps, a training loss is obtained. If the second training termination condition is not met, the model parameters of the model part other than the encoder in the anomaly detection model to be trained are adjusted based on the training loss.

[0057] On the other hand, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.

[0058] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0059] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any optional embodiment of this application.

[0060] The beneficial effects of the technical solution provided in this application are as follows:

[0061] This application provides an anomaly detection method based on feature reconstruction. This method can achieve image anomaly detection using scale-invariant (resolution-invariant) multi-level feature maps. Specifically, a trained anomaly detection model can be obtained by training a large number of normal sample images. When processing the image to be detected, the anomaly detection model extracts first feature maps of multiple levels corresponding to the image to be detected. Based on the extracted first feature maps, reconstructed feature maps corresponding to the first feature maps of each level are obtained through feature reconstruction. Since the model is trained using normal sample images, the model learns the ability to reconstruct normal image regions in the image. Theoretically, it cannot reconstruct abnormal regions in the image. However, the resolution of the reconstructed feature maps and the first feature maps is the same. Therefore, the anomaly maps corresponding to each level can be obtained directly based on the pixel value differences at corresponding positions in the reconstructed feature maps and the first feature maps of each level. Then, the obtained anomaly maps corresponding to each level can be fused. The anomaly detection result of the image to be detected is determined based on the final fused anomaly map. Since anomaly maps of different levels can express the anomalies of the image to be detected in different feature spaces / dimensions, the accuracy of the detection result can be better guaranteed based on the final anomaly map.

[0062] The solution provided in this application embodiment can achieve image anomaly detection without building a particularly complex neural network model. It has great practical application value and can be applied to various practical application scenarios with anomaly detection requirements. Moreover, it has been tested and proven to have good detection effect. Therefore, compared with anomaly detection solutions that require careful design of very complex neural networks and are extremely unfriendly to practical applications, the solution provided in this application embodiment can better meet the needs of practical applications. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0064] Figure 1 This is a schematic diagram of the structure of an anomaly detection system in an application scenario applicable to the embodiments of this application;

[0065] Figure 2 This is a schematic diagram of the structure of an anomaly detection system in another application scenario to which the embodiments of this application are applicable;

[0066] Figure 3 A flowchart illustrating an anomaly detection method based on artificial intelligence provided in this application embodiment;

[0067] Figure 4 This is a schematic diagram of the structure of an encoding / decoding module provided in an embodiment of this application;

[0068] Figure 5 Here are schematic diagrams of the structures of several anomaly detection models;

[0069] Figure 6 A schematic diagram illustrating the structure and working principle of an anomaly detection model provided in an embodiment of this application;

[0070] Figure 7 A schematic diagram illustrating the implementation principle of an anomaly detection method provided in this application embodiment;

[0071] Figure 8 A comparison chart showing the detection effects of several different anomaly detection schemes provided in the embodiments of this application;

[0072] Figure 9 A schematic diagram of an anomaly detection device based on artificial intelligence provided in an embodiment of this application;

[0073] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0074] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0075] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0076] This application addresses at least one of the problems existing in current anomaly detection technologies, such as complex implementation, poor practical application, and insufficient detection accuracy. It proposes an anomaly detection method based on artificial intelligence, which can achieve good practical application results.

[0077] Optionally, the data processing method provided in this application embodiment can be implemented based on artificial intelligence (AI) technology. For example, feature extraction and feature reconstruction of the image to be detected are achieved through a trained anomaly detection model.

[0078] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0079] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence (AI) and the fundamental way to endow computers with intelligence; its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. With the research and advancement of AI technology, it has been widely researched and applied in many fields. It is believed that with further technological development, AI will be applied in even more areas and play an increasingly important role.

[0080] Optionally, the data processing involved in the embodiments of this application can be implemented based on cloud technology. For example, the data calculations involved in the training process of the above-mentioned anomaly detection model and the data calculations involved in the anomaly detection process of the image to be detected can all be implemented based on cloud computing. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form a resource pool, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. Cloud computing refers to the delivery and use model of IT infrastructure, which means obtaining the required resources in an on-demand and easily scalable manner through the network; in a broader sense, cloud computing refers to the delivery and use model of services, which means obtaining the required services in an on-demand and easily scalable manner through the network. Unlike previous parallel distributed computing, the emergence of cloud computing will drive a revolutionary change in the entire Internet model and enterprise management model from a conceptual perspective.

[0081] It should be noted that, in the optional embodiments of this application, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0082] To better understand the solutions of the embodiments of this application, some technical terms involved in the embodiments of this application will be explained below.

[0083] AUROC stands for Area Under the Receiver Operating Curve, defined as the area under the ROC curve. It can be used as an evaluation metric for neural network models. A larger AUROC value represents better classification results. When using this metric to evaluate neural networks, the model can be evaluated at both the sample-level and pixel-level.

[0084] AP stands for Average Precision, defined as the area under the PR (precision, recall) curve. A larger AP value represents a better classification result. The model is evaluated at both the sample-level and pixel-level.

[0085] PRO stands for Per-Region-Overlap. A larger PRO value indicates better pixel-level classification results.

[0086] The Plain Vision Transformer (BVT) model is a ViT model where the resolution of the input and output feature maps remains unchanged (the feature map size remains unchanged). It can also be called columnar ViT.

[0087] The artificial intelligence-based anomaly detection method provided in this application embodiment can theoretically be applied to any application scenario that requires anomaly detection of target objects. The category of the target object can be arbitrary. For example, by using the method provided in this application embodiment, it is possible to determine whether a toothbrush is abnormal by detecting an image containing a toothbrush, to acquire an image of a circuit board, and to determine whether there are any anomalies in the circuit board, transistors, etc. by detecting the image, and it is also possible to detect where the abnormal area in the image is.

[0088] As an example, Figure 1 This paper illustrates a structural diagram of an implementation environment for an anomaly detection system applicable to embodiments of this application, as shown below. Figure 1As shown, the anomaly detection system may include an image acquisition system 11, an electronic device 12, and a database 13. The electronic device 12 is equipped with a trained anomaly detection model. The image acquisition system 11 and the electronic device 12, as well as the electronic device 12 and the database 13, can communicate via wired or wireless means. The image acquisition system 11 is used to acquire images of the target objects to be detected and send the acquired images to be detected to the electronic device 12 for processing. The electronic device 12 can execute the anomaly detection method provided in this application embodiment, and perform anomaly detection on the received images to be detected by calling the trained anomaly detection model. It can also store images with anomalies in the database 13 according to the detection results. For example, the electronic device can mark images with anomalies and mark abnormal areas, and store the marked images in the database 13 so that relevant personnel can perform corresponding processing on the target objects corresponding to the images according to the markings stored in the database. For example, in industrial product quality inspection applications, the industrial products corresponding to the marked images can be rejected.

[0089] Figure 2 A schematic diagram of the structure of an anomaly detection system in another implementation environment to which the embodiments of this application are applicable is shown, as follows: Figure 2 As shown, this anomaly detection system has multiple user terminals, such as... Figure 2 The diagram schematically shows a first terminal 21 and a second terminal 22, which are respectively servers (e.g., Figure 2 The system establishes a communication connection with the cloud server 30. Optionally, this anomaly detection system can be applied to anomaly detection of target objects in various scenarios. The cloud server 30 is a server that provides anomaly detection services. Authorized user terminals can access the cloud server via the network and send anomaly detection requests to the cloud server.

[0090] Specifically, such as Figure 2 As shown, each user terminal (first terminal 21, second terminal 22) that is connected to the cloud server 30 can be a user device in different application scenarios. When there is a detection requirement, the user terminal can send the image of the target object to be detected (such as a set of images to be detected including one or more images to be detected) to the cloud server 30 through the network. After receiving the anomaly detection request from the user terminal, the cloud server 30 can execute the method provided in the embodiment of this application, call the trained anomaly detection model to perform anomaly detection on the image to be detected, and feed back the anomaly result to the user terminal through the network.

[0091] The technical solutions of various optional embodiments provided in this application and the technical effects produced by these solutions are described below. It should be noted that the following embodiments can be referenced, borrowed, or combined with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be described again.

[0092] Figure 3 This illustration shows a flowchart of an anomaly detection method based on artificial intelligence provided in an embodiment of this application. This method can be executed by any electronic device, including a user terminal or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The user terminal can be a smartphone, tablet, laptop, desktop computer, smart voice interaction device (e.g., smart speaker), wearable electronic device (e.g., smartwatch), in-vehicle terminal, smart home appliance (e.g., smart TV), AR / VR device, etc., but is not limited to these. The user terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0093] like Figure 3 As shown in the figure, the anomaly detection method provided in this application embodiment may include the following steps S310 to S340.

[0094] Step S310: Obtain the image to be detected.

[0095] This application does not limit the method of acquiring the image to be detected. The image to be detected can be an image acquired in real time, an image stored locally, or an image received from other electronic devices. The image to be detected contains a target object, which is the object that needs to be detected for anomalies. Theoretically, the target object can be any type of object that needs to be detected for anomalies, such as a toothbrush, fruit, bottle, transistor, floor tile, etc. Through anomaly detection, it can be determined whether there is an anomaly in the target object in the image, or where the abnormal area is. It is understood that the anomaly in this application embodiment is relative. In industrial applications, an anomaly can be understood as a product being unqualified or having damage or defects. For example, floor tiles are usually smooth and have a certain regular shape under normal circumstances, so floor tiles with cracks or missing parts are a case of anomaly.

[0096] As an optional solution to this application, the above-mentioned acquisition of the image to be detected may include:

[0097] Determine the anomaly detection task type, where the anomaly detection task type is either type 1 or type 2. Type 1 is used to detect whether there is anomaly in the image, and type 2 is used to detect abnormal regions in the image.

[0098] When the anomaly detection task type is the first type, acquire the image to be detected with a resolution of the first set resolution;

[0099] When the anomaly detection task type is the second type, an image to be detected with a resolution of the second set resolution is acquired, wherein the second set resolution is greater than the first set resolution.

[0100] In this embodiment, the anomaly detection task may include a first type of anomaly detection task and a second type of anomaly detection task. The first type of anomaly detection task is a sample-level anomaly detection task used to detect whether anomalies exist in the image. The second type of anomaly detection task is a pixel-level anomaly detection task used to detect abnormal regions in the image. Accordingly, the anomaly detection result of the first type of anomaly detection task is either that an anomaly exists in the image or that no anomaly exists in the image. The anomaly detection result of the second type of anomaly detection task is either that an abnormal target region / abnormal region exists in the image or that no abnormal region exists in the image.

[0101] In practical applications, different application scenarios usually have different detection requirements. Considering that when using neural networks for image-based task processing, the resolution of the image input into the model will also affect the accuracy of the final processing result, in order to better meet different anomaly detection requirements and improve the accuracy of detection results, in actual implementation, this application embodiment can obtain the image to be detected at a set resolution corresponding to the current anomaly detection task type, that is, the sample-level anomaly detection task corresponds to the first set resolution, and the pixel-level anomaly detection task corresponds to the second set resolution.

[0102] The process of acquiring the image to be detected at a first or second set resolution can be achieved by acquiring an image of the corresponding resolution in real time according to the set resolution. If the resolution of the acquired image is not equal to the set resolution corresponding to the task type, the image to be detected can be converted into an image of the corresponding set resolution according to the specific task type, and subsequent processing can be performed based on the converted image.

[0103] Sample-level anomaly detection aims to determine whether anomalies exist in the overall image. Compared to the accuracy of detection at a specific location within the image, sample-level anomaly detection focuses more on the accuracy of the overall image assessment. Pixel-level anomaly detection, on the other hand, requires detecting anomalous regions within the image, focusing on the detection results at specific locations, making it a more fine-grained anomaly detection task. Therefore, by configuring a relatively larger resolution for pixel-level anomaly detection (the second set resolution is greater than the first set resolution), the anomaly detection model can acquire more pixel information in the image during feature extraction and reconstruction in subsequent steps S320, thereby improving the accuracy of anomaly region detection. Conversely, by configuring a relatively smaller resolution for sample-level anomaly detection, the amount of data to be processed can be reduced without significantly affecting the overall accuracy of the image detection results, thus improving detection efficiency.

[0104] Optionally, in practical application scenarios, if it is necessary to both detect whether there are abnormalities in the image and, if there are abnormalities, further detect abnormal areas in the image, an image to be processed at a third set resolution can be obtained. The third set resolution can be equal to the first set resolution or the second set resolution, or it can be other set values, such as a set value between the first set resolution and the second set resolution.

[0105] The specific values ​​of the first and second set resolutions mentioned above are not uniquely limited in this application embodiment and can be determined based on empirical and / or experimental values. Optionally, the first set resolution can be 224*224, and the second set resolution can be 288*288 or 384*384. After obtaining the image to be detected at the corresponding resolution according to the anomaly detection task type, step S320 can be proceeded.

[0106] Step S320: Use the trained anomaly detection model to extract and reconstruct features from the image to be detected, such as... Figure 3 As shown, this step may include a feature extraction step S310 and a feature reconstruction step S320, as detailed below:

[0107] Step S321: Perform feature extraction on the image to be detected to obtain first feature maps at multiple levels;

[0108] Step S322: Reconstruct the first feature maps of at least some of the multiple layers based on the intermediate feature maps to obtain the reconstructed feature maps corresponding to the first feature maps of each of the multiple layers. The intermediate feature maps include at least the first feature map of the last layer in the first feature maps of the multiple layers, and the intermediate feature maps may also include the first feature maps of other layers.

[0109] In this embodiment, the first feature map of at least some of the aforementioned layers does not include the first feature map of the last layer. In this embodiment, the intermediate feature map used for reconstruction includes at least the first feature map of the last layer, and the reconstructed feature map is the reconstructed feature map corresponding to the first feature maps of one or more layers other than the last layer. Optionally, the intermediate feature map may include the first feature maps of each layer in multiple layers, and may also include the image to be detected itself. Besides obtaining the reconstructed feature maps corresponding to the first feature maps of one or more layers through feature reconstruction, it can also reconstruct the reconstructed image corresponding to the image to be detected. It is understood that the terms "first" in the first feature map and "reconstructed" in the reconstructed feature map in this embodiment are used to distinguish between the feature map obtained by feature extraction from the image and the reconstructed feature map obtained by feature map reconstruction based on the extracted feature map. In some embodiments below, the first feature map may also be referred to as the original feature map.

[0110] Optionally, the resolution of the first feature map and the reconstructed feature map is the same for all levels. Optionally, the resolution of the first feature map and the reconstructed feature map can also be different for different levels, while the resolution of the first feature map and the reconstructed feature map corresponding to the same level is the same.

[0111] In this embodiment, the anomaly detection module may include an encoder and a decoder. The encoder is used to extract features from the image to be detected, and the decoder then performs feature reconstruction based on the feature maps extracted by the encoder to obtain the reconstructed feature maps. The level of the feature maps extracted by the encoder refers to the depth of the feature maps. The encoder may include one or more cascaded encoding modules (feature extraction modules). The input of each subsequent encoding module is the output of the preceding encoding module, i.e., the feature map extracted by the preceding encoding module. The first feature map of the first level is obtained by feature extraction based on the image to be detected, and the first feature map of the second level is obtained by feature extraction based on the feature map of the first level. The decoder may include one or more cascaded decoding modules. Each encoding module and each decoding module may each include one or more cascaded feature processing layers. For the encoding module, the feature processing layer is a feature extraction layer; for the decoding module, the feature processing layer is a feature reconstruction layer. The network structures of the feature extraction layer and the feature reconstruction layer may be the same or different. The first feature maps of multiple levels corresponding to the image to be detected, and the reconstructed feature maps corresponding to the first feature maps of each level, may be obtained in the following ways:

[0112] The image to be detected is input into the encoder, and features are extracted sequentially through each encoding module of the encoder to obtain the first feature map of a level corresponding to each encoding module. The input of the next encoding module is the first feature map output by the previous encoding module.

[0113] Based on the intermediate feature map, the first feature map of each level in at least some levels is reconstructed by each decoding module in the decoder to obtain the reconstructed feature map corresponding to the first feature map of each level in at least some levels.

[0114] Optionally, during feature reconstruction, the intermediate feature maps used may include at least one level of first feature maps, and may also include the image to be detected, or may not include the image to be detected. The reconstructed feature map to be reconstructed may be a reconstructed feature map corresponding to the first feature maps of at least one level other than the last level, such as a reconstructed feature map corresponding to one or more levels of first feature maps other than the last level, or a reconstructed feature map corresponding to the first feature maps of all levels other than the last level.

[0115] In this embodiment, optionally, the first feature map of each level can be the output feature of an encoding module, and the reconstructed feature map corresponding to the first feature map of each level can be the output feature of a decoding module. The number of encoding modules and decoding modules can be the same or different; that is, the encoder and decoder can have a symmetrical structure or a non-symmetrical structure. Optionally, the encoding module corresponding to the first feature map of the first level can also be different from the encoding modules of the first feature maps of other levels.

[0116] As an example, suppose the encoder includes one initial feature extraction module (such as a convolutional feature extraction module) and four cascaded encoding modules, and the decoder includes four cascaded decoding modules. The input to the initial feature extraction module corresponds to the image to be detected. The initial feature extraction module can extract the first feature map F0 of the first layer. After F0 is input to the second encoding module, it is passed through the first encoding module to obtain the first feature map F1 of the second layer. After F1 is input to the second encoding module, it is passed through the second encoding module to obtain the first feature map F2 of the third layer, and so on. After F2 is input to the third encoding module, it is obtained the first feature map F3 of the fourth layer. After F3 is input to the fourth encoding module, it is obtained the first feature map F4 of the fifth layer. Then, the encoder in this example can extract feature maps F0 to F4 of five layers corresponding to the image to be detected.

[0117] After obtaining the first feature maps of multiple levels, the first feature maps of at least one level, including the last level's first feature map F4, can be input into the decoder. The first decoding module of the decoder then reconstructs the reconstructed feature map corresponding to F3. , The input is fed into the second decoding module, which reconstructs the feature map corresponding to F2. And so on, based on The reconstructed feature map corresponding to F1 is obtained through the third decoding module. ,based on The reconstructed feature map corresponding to F0 is obtained through the fourth decoding module. .

[0118] To improve model performance and ensure the reconstruction results of feature maps, as an optional solution in this application, the trained anomaly detection model may further include a feature fusion unit that connects the encoder and decoder. The above-mentioned reconstruction of the first feature map of each level in at least some layers based on the intermediate feature map to obtain the reconstructed feature map corresponding to the first feature map of each level in at least some layers may include:

[0119] The first feature maps of each level in the intermediate feature maps are fused by a feature fusion processor to obtain a fused feature map. The fused feature map is then input into a decoder, and the decoder reconstructs the reconstructed feature map corresponding to each feature map in at least some levels of the first feature maps.

[0120] It should be noted that in actual implementation, feature map reconstruction can also use only the first feature map of the last level among multiple levels of first feature maps. In this case, the anomaly detection model may not include a feature fusion unit, that is, the first feature map of the last level can be directly input into the decoder. Alternatively, even if reconstruction is only based on the first feature map of the last level, the model can still include a feature fusion unit. In this case, the feature fusion unit can be used to perform feature transformation on the first feature map of the last level to obtain features that meet the input requirements of the decoder and have better feature representation capabilities.

[0121] The structure of the feature fusion processor is not limited in this application embodiment. Optionally, in order to reduce the complexity of the model, the feature fusion processor may include one or more cascaded linear layers. The linear layers perform linear transformation on the feature maps to be fused to obtain a more compact fused feature map.

[0122] The anomaly detection model is obtained by training a neural network model (the anomaly detection model to be trained) using machine learning. Specifically, the neural network model can be continuously trained using a training image set until a preset training termination condition is met to obtain a trained anomaly detection model. In this embodiment, the training image set is an image set containing multiple sample images without anomalies (i.e., non-abnormal sample images). In other words, the training samples of the anomaly detection model are all positive samples / normal images, and the sample objects in the sample images are all objects without anomalies.

[0123] Understandably, during the training phase, the anomaly detection model processes sample images, while during the application / inference phase, the trained anomaly detection model processes the image to be detected. However, regardless of the training or inference phase, the process of feature extraction and reconstruction using this model is the same. Specifically, during the training phase, the anomaly detection model is used to extract features from the sample images, obtaining multiple levels of sample feature maps corresponding to the sample images. Reconstruction can then be performed based on the intermediate sample feature maps corresponding to the sample images, resulting in reconstructed sample feature maps corresponding to each level of sample feature maps. In this embodiment of the application, the training loss of the anomaly detection model to be trained can characterize the difference between the sample feature map and the reconstructed sample feature map corresponding to each of the above M levels of the sample image. Optionally, the training loss can be calculated based on the anomaly map of each of the M levels of the sample image. For example, it can be determined based on the sum or mean of the pixel values ​​in the anomaly map of each of the M levels of the sample image. For example, for a sample image, the sum or mean of the pixel values ​​of all positions, i.e., all pixels, in the anomaly map of the M levels of the sample image can be used as the training loss of the sample image. The sum or mean of the training losses of multiple sample images can be used as the training loss of the model.

[0124] As an alternative, to ensure the performance of the trained anomaly detection model and reduce the complexity of the training process, the model's encoder can be pre-trained. After obtaining the pre-trained encoder, its model parameters can be fixed, and then the other parts of the model can be trained to obtain the trained anomaly detection model. Specifically, the trained anomaly detection model can be obtained through the following methods:

[0125] Obtain the first training set, which includes multiple labeled training images. The label of a training image represents the true category of that training image.

[0126] The first neural network model is trained on the first training set until the first training termination condition is met. The first neural network model includes an encoder of the anomaly detection model to be trained and a classifier cascaded with the encoder. The input and output of the first neural network model are the training image and the predicted category of the training image, respectively. The training loss of the first neural network model represents the difference between the true category and the predicted category of each training image.

[0127] Obtain a second training set, which includes multiple non-abnormal sample images, wherein the multiple sample images include images of sample objects of multiple categories;

[0128] The encoder that meets the first training termination condition is taken as the trained encoder. The model parameters of the trained encoder are fixed. The part of the anomaly detection model to be trained, excluding the encoder, is trained based on the second training set until the second training termination condition is met, and the trained anomaly detection model is obtained.

[0129] Specifically, the anomaly detection model to be trained can be repeatedly trained based on the second training set until the second training termination condition is met, resulting in a trained anomaly detection model. This training operation includes the following steps:

[0130] Each sample image is input into the anomaly detection model to be trained, and the M-level sample feature maps corresponding to each sample image and the reconstructed sample feature maps corresponding to the M-level sample feature maps are obtained.

[0131] Based on the differences between the M-level sample feature maps corresponding to each sample image and the reconstructed sample feature maps corresponding to the M-level sample feature maps, a training loss is obtained. If the second training termination condition is not met, the model parameters of the model part other than the encoder in the anomaly detection model to be trained are adjusted based on the training loss.

[0132] In this scheme, the pre-training process of the encoder can be based on classification task training. The first training set consists of training images with known true categories (which can be the true types of objects contained in the image). The training images are input into the first neural network model. The encoder to be trained can extract the encoded features of the training images. Based on the encoder features, the classifier can predict the predicted category of the image. If the first training termination condition is not met (such as the number of training iterations reaching a set number or the training loss function corresponding to the first neural network model converging), the training loss of the first neural network model can be calculated based on the difference between the true category and the predicted category of each training image in the first training set. Based on the training loss, the model parameters of the encoder and the classifier are adjusted, and the above training process is repeated based on the training images to at least satisfy the first training termination condition, thus obtaining a pre-trained encoder.

[0133] Afterwards, the model parameters of the pre-trained encoder can be fixed, and the other parts of the anomaly detection model to be trained (fusion unit, encoder) can be trained. Specifically, sample images without anomalies can be input into the anomaly detection model to be trained. The model can then perform the aforementioned feature extraction and feature reconstruction operations on the sample images to obtain sample feature maps of multiple levels corresponding to the sample images, as well as reconstructed sample feature maps corresponding to at least some levels of the sample feature maps. Since the sample images are all positive samples without anomalies, the sample feature maps and reconstructed sample feature maps corresponding to each level should be very similar, theoretically they should be basically the same. Based on this, we can obtain the sample anomaly maps corresponding to each of the M levels by calculating the image differences between the sample feature maps and reconstructed sample feature maps corresponding to each level of the sample image. The sum of pixel values ​​at all positions in the M sample anomaly maps can be used as the training loss of the sample image, or the sum of pixel values ​​at each position in each anomaly map can be calculated separately, and the mean of the sum of pixel values ​​corresponding to the M anomaly maps can be used as the training loss of the sample image. The sum or mean of the training losses of each sample image can be used as the training loss of the model. Based on the training loss of the model, the gradient descent algorithm is used to optimize and adjust the model parameters other than the encoder model parameters. The adjusted model is then trained using each sample image until a well-trained anomaly detection model that meets the second training termination condition is obtained.

[0134] In this embodiment of the application, the second training set may include images of sample objects of at least one category. Optionally, the second training set may include sample images of sample objects of multiple different categories. By using images of sample objects of multiple different categories to train the model, the trained anomaly detection model can be applied to anomaly detection tasks of multiple categories. Moreover, as proven by testing, the anomaly detection method provided in this embodiment of the application has high accuracy in anomaly detection tasks of multiple different categories.

[0135] Since the anomaly detection model is trained on a large number of positive samples without anomalies, it should learn the experience / ability of how to reconstruct normal regions in an image. It can be assumed that the model cannot correctly reconstruct abnormal regions in an image. In other words, for regions without anomalies in an image, the reconstructed region corresponding to that region in a certain level of the reconstructed feature map obtained by the model should be very similar to the region corresponding to that region in the first feature map of that level. Conversely, for regions with anomalies in an image, since the model does not have the ability to reconstruct abnormal regions or its ability to reconstruct abnormal regions is very poor, the region corresponding to the abnormal region in the reconstructed feature map should be significantly different from the region corresponding to that region in the first feature map.

[0136] Based on the above principles, after obtaining the first feature maps and reconstructed feature maps of each level of the image to be detected through the trained anomaly detection model, the anomaly detection result of the image to be detected can be determined based on the differences between the first feature maps and reconstructed feature maps of at least some of these levels. Specifically:

[0137] Step S330: Based on the first feature maps of M levels and the reconstructed feature maps corresponding to the first feature maps of M levels, obtain the anomaly map corresponding to each level in the M levels. The anomaly map corresponding to a level is used to characterize the pixel difference between the first feature map and the reconstructed feature map of that level. The M levels belong to at least some of the above-mentioned levels.

[0138] Step S340: Identify the anomaly detection results of the image to be detected based on the anomaly maps corresponding to M levels.

[0139] Optionally, the M levels may include each of the at least some of the levels, that is, the M levels are the at least some of the levels, or the M levels may be some of the at least some of the levels.

[0140] Specifically, for any reconstructed feature map corresponding to a certain level, an anomaly map corresponding to that level can be obtained based on the image difference (pixel information difference) between the first feature map and the reconstructed feature map corresponding to that level. Optionally, the pixel value at a certain position in the anomaly map corresponding to a certain level can be obtained based on the difference between the pixel value at that position in the reconstructed feature map and the first feature map of that level. The pixel value at that position in the anomaly map represents the probability that there is an anomaly at that position in the image to be detected. The larger the pixel value, the higher the probability of an anomaly.

[0141] Optionally, both the first feature map and the reconstructed feature map of each level include multiple channels, and the number of channels in the first feature map of a level is the same as the number of channels in the reconstructed feature map. For each of the above M levels, the anomaly map corresponding to that level is obtained in the following way:

[0142] For each identical position in the first feature map and the reconstructed feature map of this level, the pixel values ​​of that position in multiple channels of the first feature map of this level are combined to obtain the original feature vector corresponding to that position, and the pixel values ​​of that position in multiple channels of the reconstructed feature map corresponding to the first feature map of this level are combined to obtain the reconstructed feature vector corresponding to that position.

[0143] For each of the above identical positions, determine the degree of difference between the original feature vector and the reconstructed feature vector corresponding to that position, and use this degree of difference as the pixel value of that position in the anomaly map corresponding to that level.

[0144] Since the resolution of the first feature map and the corresponding reconstructed feature map of a level is the same, the number of positions (i.e., pixels, or feature points) in the feature maps of each channel of the first feature map and the reconstructed feature map of each level is also the same. The pixel values ​​(also called feature values) in the first feature map are obtained by feature extraction from the image to be detected. The reconstructed feature map is obtained by feature reconstruction based on the first feature map of the image, i.e., the original feature map. Therefore, for each position in the original feature map of a level, the pixel values ​​of that position in the feature maps of multiple channels included in the original feature map can be combined (e.g., concatenated) to obtain the original feature vector of that position. Similarly, the pixel values ​​of that position in the feature maps of multiple channels included in the reconstructed feature map can be combined (e.g., concatenated) to obtain the reconstructed feature vector of that position. Then, based on the degree of difference between the original feature vector and the reconstructed feature vector of that position, the pixel value of that position in the anomaly map of that level can be obtained. In this way, the pixel values ​​of each position in the anomaly map corresponding to a level can be calculated, thus obtaining the anomaly map. The resolution of the anomaly map, the resolution of the reconstructed feature map, and the resolution of the original feature map of a level are all the same.

[0145] The specific method for determining the degree of difference at each location is not limited in the embodiments of this application. Optionally, the degree of difference at a location can be obtained by calculating the similarity between the original feature vector and the reconstructed feature vector corresponding to that location. For example, the similarity value can be a numerical value from 0 to 1, with a higher value indicating a higher similarity. The degree of difference can be obtained by subtracting the similarity value from 1. The method of calculating the similarity is not limited in this application. Alternatively, the degree of difference at a location can be obtained by calculating the mean absolute error (MAE) or mean-square error (MSE) between the original feature vector and the reconstructed feature vector corresponding to that location. In the anomaly map, a larger pixel value at a location indicates a higher probability that there is an anomaly at that location in the image to be detected.

[0146] After obtaining the anomaly maps corresponding to the above M levels of the image to be detected, the anomaly detection results of the image to be detected can be identified based on the anomaly maps corresponding to these M levels. For example, the anomaly detection results can be identified based on the pixel values ​​at each position in the first feature map corresponding to the M levels.

[0147] Optionally, the anomaly detection results of the image to be detected, based on the anomaly maps corresponding to the M levels, may include:

[0148] The anomaly maps corresponding to M levels are fused to obtain the final anomaly map; the anomaly detection results of the image to be detected are identified based on the final anomaly map.

[0149] Since feature maps at different levels can correspond to different feature spaces and express information in different dimensions of an image, a more accurate final anomaly map can be obtained by fusing anomaly maps from multiple levels. Based on this final anomaly map, the anomaly detection result of the image to be detected can be determined. The anomaly detection result can be at least one of the following: whether an anomaly exists in the image to be detected, or whether an anomaly region exists in the image to be detected. That is, depending on the actual detection requirements, anomaly detection can be detecting the presence of an anomaly (sample-level anomaly detection), or detecting anomaly regions in the image (pixel-level anomaly detection, or anomaly localization).

[0150] In this embodiment of the application, the first feature map of at least some levels includes the first feature map of each level other than the first feature map of the last level in the first feature map of the plurality of levels, and the M levels include each level other than the first level in the plurality of levels.

[0151] In this optional scheme, feature map / image reconstruction can be performed based on intermediate feature maps (first feature maps of some or all layers) to obtain reconstructed feature maps corresponding to the first feature maps of some or all layers except the first feature map of the last layer. For example, reconstructed feature maps corresponding to the first feature maps of each layer except the last layer can be obtained. Optionally, reconstructed feature maps (reconstructed images) corresponding to the image to be detected can also be obtained. After obtaining these reconstructed feature maps, the first feature maps of each layer except the first feature map of the first layer and its corresponding reconstructed feature map and the reconstructed feature map of the corresponding layer can be used to calculate the abnormal feature maps corresponding to each layer except the first layer. That is, when determining the abnormal detection result of the image to be detected, the first feature map of the first layer and its corresponding reconstructed feature map can be omitted, or the image to be detected and the reconstructed image can be omitted.

[0152] As an example, suppose the first feature maps of all levels include feature maps F0, F1, F2, F3, and F4. The first feature maps F0 to F4 of all levels can be fused, and the fused feature map is input into the decoder to obtain the reconstructed feature maps corresponding to F0 to F3 respectively. F3 and The corresponding anomaly diagrams A3 and F2 are The corresponding anomaly diagrams A2 and F1 are... The corresponding anomaly image A1 is obtained by adding or summing the pixel values ​​at the same positions in A1 to A3. Based on the pixel value information of the anomaly image A, it can be determined whether there is an anomaly in the image to be detected, or where the abnormal regions in the image to be detected are located.

[0153] Since the reconstruction accuracy between the first feature map and the corresponding reconstructed feature map at different levels is likely to be different, and considering that the first feature map at the first level is obtained by feature extraction from the image to be detected, the reconstructed image at this level is the final image obtained after multiple feature extractions and multiple reconstructions based on the extracted features, compared to the reconstructed feature maps at other levels and their corresponding first feature maps (e.g., the first feature map F3 and its corresponding reconstructed feature map), the accuracy of the reconstruction is likely to differ. The first feature map and the corresponding reconstructed feature map of the first level have undergone more feature transformations. More feature transformations may also lead to a larger deviation between the reconstructed map and the image to be detected. Therefore, when calculating anomaly maps, the anomaly maps corresponding to the image to be detected and the reconstructed image can be omitted.

[0154] Of course, when there are many levels in the above-mentioned multiple levels, when calculating the anomaly map, in addition to excluding the first feature map of the first level in the first feature map of multiple levels, the first feature map of other shallower levels, such as the first feature map of the second level, can also be excluded.

[0155] Correspondingly, when training the anomaly detection model using normal sample images, the training loss of the model based on the anomaly map corresponding to the sample image can also be calculated without using the first-level sample feature map of the sample image and the anomaly map corresponding to the reconstructed sample feature map.

[0156] The above-described scheme provided in the embodiments of this application was tested on some public datasets. The test results show that, compared with the method of calculating the anomaly map by using the first feature map of the first level and the reconstructed feature map corresponding to the first feature map, the method of calculating the anomaly map without using the first feature map of the first level and the reconstructed feature map corresponding to the first level has a better anomaly detection effect.

[0157] Of course, as another alternative, the first feature map of all levels and the corresponding reconstructed feature map can be used to calculate the anomaly map for each level in all levels. Alternatively, the anomaly map for one level can be calculated using the image to be detected and the reconstructed image.

[0158] Optionally, after obtaining the anomaly maps corresponding to M levels, the final anomaly map can be obtained by fusing the anomaly maps corresponding to the M levels in the following way:

[0159] By adding the pixel values ​​at the same position in the anomaly maps corresponding to M levels, the pixel value at the corresponding position in the final anomaly map is obtained.

[0160] The above-mentioned anomaly detection results based on the final anomaly map of the image to be detected include at least one of the following:

[0161] The final anomaly map is fused by region, where each region has a set size. The fused pixel values ​​of each region are used as the anomaly score of that region. Based on the anomaly scores of each region, it is determined whether the target object has an anomaly.

[0162] The pixel value at each location in the final anomaly image is used as the anomaly score for that location. If there are locations in the final anomaly image with an anomaly score not less than the second threshold, the image regions corresponding to the locations in the image to be detected with an anomaly score not less than the second threshold in the final anomaly image are determined as anomaly regions.

[0163] For sample-level anomaly detection, an image-level anomaly score is needed to assess whether the image to be detected has anomalies. For pixel-level anomaly detection, pixel-level anomaly scores (i.e., anomaly scores at each location) are needed to locate abnormal regions in the image to be detected. In this embodiment, after obtaining the M anomaly maps corresponding to the above M levels, since the pixel values ​​at each location in the anomaly map represent the probability of anomalies at the corresponding location in the image to be detected, the pixel values ​​at corresponding locations in the M anomaly maps can be added together to obtain the final anomaly map of the image to be detected. Optionally, assuming M=3, the size of the anomaly map is H*W, where H represents the height of the image and W represents the width of the image, then by adding the three pixel values ​​at each corresponding location in the three anomaly maps, a final anomaly map of size H*W is obtained. For example, the sum of the three pixel values ​​at each corresponding location or the average of the three pixel values ​​can be used as the pixel value at each corresponding location in the final anomaly map.

[0164] After obtaining the final anomaly map, the anomaly detection result of the image to be detected can be determined based on the pixel values ​​in the anomaly map. When anomaly detection is a sample-level anomaly detection task, since the goal is to determine whether there are anomalies in the entire image, the pixel value of a single point in the anomaly map cannot accurately represent the anomaly situation of the entire image. For example, the large pixel value (outlier) of one or several points in the anomaly map may be due to noise or other reasons in the image to be detected, even if the image itself does not have any anomalies. In order to ensure the overall anomaly detection result, after obtaining the final anomaly map, the pixel values ​​of the anomaly map can be fused by dividing it into regions. The region size of each region is a preset size, which is smaller than the size / resolution of the anomaly map. For example, the preset size can be used as the size of the average pooling (i.e., the size of the pooling kernel). Average pooling is performed on the final anomaly map according to this size and the preset step size. The average pooling result of each image region of the corresponding size in the anomaly map is used as the new pixel value of each pixel in that image region, which is the anomaly score corresponding to that image region. Alternatively, the final anomaly map can be simply divided into multiple image regions according to the preset size, and then the pixel values ​​in each image region are averaged or summed to obtain the anomaly score corresponding to each image region. Then, the presence of anomalies in the image to be detected can be determined based on the anomaly scores of each region. For example, if the maximum anomaly score among multiple anomaly scores is greater than the first threshold, the image to be detected can be considered to have anomalies.

[0165] For pixel-level anomaly detection tasks, since the size of the final anomaly map is equal to the size of the image to be detected, and the pixel value of each pixel in the final anomaly map is the anomaly value at the corresponding position in the image to be detected, we can determine which pixels are anomalies based on whether the pixel value of each pixel in the final anomaly map is not less than the second threshold, and then determine the image region corresponding to the identified anomaly point as the anomaly region.

[0166] This application provides an anomaly detection method based on feature reconstruction. This method uses a trained anomaly detection model to extract first feature maps at multiple levels from the image to be detected, and then performs feature reconstruction based on these first feature maps to obtain a reconstructed feature map. Since the anomaly detection model is trained using non-abnormal sample images, anomaly detection can be achieved based on the difference between the first feature map at least one level and the corresponding reconstructed feature map of the image to be detected. Optionally, this method can use the final anomaly map obtained by fusing anomaly maps from multiple levels (M levels) to determine the anomaly detection result of the image to be detected, which can effectively improve the accuracy of the detection result. Optionally, in the method provided by this application, the resolution of the feature maps in the encoding and decoding parts of the anomaly detection model is constant, and it is a columnar neural network model. This model has a simple structure, and compared to the complex model structure using pyramid networks, the model provided by this application can be well applied to various practical anomaly detection scenarios and can guarantee good detection accuracy, better meeting practical needs.

[0167] The specific neural network structure of the anomaly detection model is not uniquely limited in the embodiments of this application. As an optional solution, both the encoder and decoder of the anomaly detection model adopt a neural network structure based on the Naive Visual Transformation (ViT) network, with each encoding module including at least one cascaded feature extraction layer and each decoding module including at least one cascaded feature reconstruction layer. Optionally, the neural network structure of the anomaly detection model satisfies at least one of the following:

[0168] Each feature extraction layer and each feature reconstruction layer corresponds to a Transformer encoding module / unit;

[0169] The Transformer encoding module described above does not include a normalization layer;

[0170] The inputs and outputs of the Transformer encoding module described above do not include classification token features;

[0171] The feature fusion unit of an anomaly detection model can include one or more cascaded linear layers.

[0172] In this optional scheme, the encoding module and the decoding module can adopt the same network structure, or each encoding module and each decoding module except the first encoding module can adopt the same network structure. Each encoding / decoding module includes one or more cascaded Transformer encoding modules. One Transformer encoding module corresponds to a basic encoding unit in the Naive ViT network. The Transformer encoding module can directly adopt a basic encoding unit in the Naive ViT network. As one embodiment of this application, each feature extraction layer / feature reconstruction layer in the encoding / decoding module of the anomaly detection model can adopt a network structure that improves the basic Transformer encoding unit of the ViT network. Specifically, the network structure can discard the normalization layer / standardization layer in the basic encoding unit of the ViT network. Optionally, the input and output of the network structure may not include classification token features.

[0173] The improved network structure described above can reduce the structural complexity of the anomaly detection model. Tests have shown that, compared to directly using the existing Transformer encoding module of the ViT network as the feature extraction / reconstruction layer, using the Transformer encoding module that discards the normalization layer can achieve better detection results. Compared to using features with the classification token dimension in the input and output, discarding the classification token dimension in the entire model structure also results in better detection results.

[0174] Optionally, when using an anomaly detection model based on a naive ViT network, the above-mentioned input of the image to be detected into the encoder, and feature extraction is performed sequentially through each encoding module of the encoder, including:

[0175] The image to be detected is divided into multiple image patches according to the target size.

[0176] For each image block, the image feature vector of the image block is obtained based on the pixel information of the image block, the position feature vector of the image block is obtained according to the position of the image block in the image to be detected, and the initial feature vector of the image block is obtained by fusing the image feature vector and the position feature vector of the image block;

[0177] The initial feature vectors of multiple image patches are used as input to the encoder, and features are extracted sequentially by each encoding module in the encoder.

[0178] The above-mentioned feature fusion method fuses the first feature maps of each level in the intermediate feature maps to obtain a fused feature map, including:

[0179] The first feature maps of each level in the intermediate feature maps are concatenated by a feature fusion processor according to the channel dimension, and the concatenated feature maps are then transformed to obtain a fused feature map with the number of channels equal to the number of output channels of the encoding module.

[0180] The specific architecture of the aforementioned naive ViT network may include, but is not limited to, a small-scale ViT-Small network, or a ViT model of other scales, or a structure that has been improved based on the naive ViT network structure.

[0181] As an example, Figure 4 This illustration shows a network structure diagram of an encoding / decoding module according to an embodiment of this application. The encoding / decoding module in this example may include three Transformer encoding modules (L×3 in the diagram represents three identical stacked / cascaded Transformer encoding modules). The structure of one Transformer encoding module is... Figure 4 The structure within the rectangular frame, such as Figure 4 As shown, a Transformer encoding module / unit may include a multi-head attention layer and a multilayer perceptron. The input of the first Transformer encoding module in the first encoding module corresponds to the initial feature vector of the image to be detected. This initial feature vector includes the initial feature vectors of each image patch after the image to be detected is divided, excluding the initial feature vector of the classification token. The initial feature vector of each image patch is obtained by adding the pixel features (image feature vector) and the positional features of the image patch. The feature map output by the first Transformer encoding module is the encoded feature corresponding to each image patch. This encoded feature is input to the second Transformer encoding module for further feature extraction. The encoded feature output by the second Transformer encoding module is then subjected to further feature extraction by the third Transformer encoding module to obtain the output feature of the encoder's first encoding module, which is the first feature map of the first level. This first feature map of the first level is the input of the second encoding module. After obtaining the first feature maps of multiple levels through each encoding module, the first feature maps of each level can be fused by a feature fusion unit. The fused, more compact feature map is input to the decoder, which reconstructs the reconstructed feature maps of each level.

[0182] As an optional solution in this application, when dividing the image to be detected into multiple image blocks according to a target size, the target size can be determined in the following way:

[0183] Determine the anomaly detection task type, which is either type 1 or type 2. Type 1 is used to detect whether there are anomalies in the image, while type 2 is used to detect abnormal regions in the image.

[0184] If the task type is type 1, the first dimension is determined as the target dimension; if the task type is type 2, the second dimension is determined as the target dimension, and the second dimension is smaller than the first dimension.

[0185] As described above, the resolution of the image to be detected affects the accuracy of the detection results for different anomaly detection tasks. Similarly, the resolution of image patches also affects the accuracy of the final detection results. Compared to sample-level anomaly detection tasks, pixel-level anomaly detection tasks are more susceptible to changes in the detail information of the input image. Based on this, this application proposes an optional scheme for setting different image patch partitioning sizes for different anomaly detection tasks. The target size corresponding to the image patch in a pixel-level anomaly detection task can be smaller than the target size in a sample-level anomaly detection task. The specific values ​​of the first and second sizes can be determined based on experimental or empirical values. For example, the first size can be 16*16, and the second size can be 8*8.

[0186] In current technologies, naive ViT networks have never been explored in the field of anomaly detection. Anomaly detection schemes based on neural network models typically employ hierarchical pyramid networks as encoders / decoders at the encoder and / or decoder levels, such as... Figure 5 The diagrams (a), (b), and (c) show the structures of three neural network models. Researchers in the field seem to agree that hierarchical features are crucial for accurately modeling anomaly locations, but neural network models with multi-scale features require carefully designed complex structures, which are not very user-friendly for practical applications.

[0187] To better meet the increasing demand for anomaly detection and to make deep learning-based anomaly detection solutions highly valuable in industrial anomaly detection scenarios, this application summarizes reconstruction-based unsupervised anomaly detection models and abstracts the concept of a meta-anomaly detection (MetaAD) model, such as... Figure 5 The model architecture shown in (d) includes an encoder (Encoder). Figure 5 In φ E Feature Fuder Fuser Figure 5 In F The three parts are ( ) and decoder (Decoder) Figure 5 In φ D).

[0188] In the practical implementation of the meta-anomaly detection model, at least one of the encoder or decoder can adopt a pyramid network, that is, a network structure that extracts multi-scale features. Neither the encoder nor the decoder needs to adopt a pyramid network structure. In order to enable the deep learning-based anomaly detection scheme to be practically applied in actual industrial anomaly detection scenarios and improve the practical application value of the detection scheme, this application proposes an unsupervised anomaly detection model based on a columnar network, on the basis of the meta-anomaly detection concept. Here, the columnar network can be interpreted as the resolution of the features extracted by the encoder and the features reconstructed by the decoder being constant, that is, neither the encoder nor the decoder network structure adopts a multi-scale feature extraction network.

[0189] To better understand and illustrate the solutions provided in the embodiments of this application and their practical value, the following uses an anomaly detection model based on a naive columnar ViT network as an example to further explain the solutions provided in this application.

[0190] Specifically, in an optional embodiment of this application, an unsupervised anomaly detection model based on naive ViT, ViTAD (Vision Transformer Anomaly Detection), is proposed. As mentioned above, the encoder and decoder of this model can contain only the most basic columnar (feature map scale invariant) ViT structure, and the fusion generator (Fuser) can contain several linear layers. This naive structure not only greatly alleviates the adaptation pressure of hierarchical models, but also achieves better results than other model structures (models based on pyramid networks) on standard evaluation datasets. It is evident that complex pyramid structures are not essential for anomaly detection models; on the contrary, with proper design, simple columnar networks can also achieve excellent results. Furthermore, in the optional embodiments of this application, the feature fusion generator is optional rather than mandatory.

[0191] As an alternative, Figure 6 This illustration shows a structural diagram of an anomaly detection model based on Naive ViT provided in an embodiment of this application, as follows: Figure 6 As shown, the anomaly detection model includes an encoder. E and decoder D and feature fusion that connects the encoder and decoder. F The encoder consists of several encoding modules, and the decoder consists of several decoding modules. Both the encoder and decoder have a symmetrical structure, and the encoding and decoding modules employ the same neural network structure based on the columnar ViT network structure. Figure 6In the example, the encoder and decoder use a symmetrical network structure, each consisting of four cascaded modules, namely, sequentially cascaded encoding modules. , , and and the cascaded decoding modules , , and In practical implementation, the encoder may also include the first encoding module. The initial feature extraction module, in its previous stage, takes as input the first feature map F0 of the first level obtained by the initial feature extraction module from the input image (sample image or image to be detected). The reconstructed feature map corresponding to F0 is based on... The output reconstructed feature map is used for reconstruction. The structures of the initial feature extraction module and the decoding module used to obtain the reconstructed feature map corresponding to F0 are not limited in this embodiment and can be the same as or different from the encoding module. The following description mainly focuses on the encoder. , , and and decoding module , , and The description is as follows.

[0192] For a naive ViT network, the Transformer encoder can include multiple stacked identical Transformer layers / modules / units. In this optional embodiment, both the encoder and decoder can include a Transformer encoder with 12 stacked Transformer units. Optionally, both the encoder and decoder can employ... Figure 4 The network structure shown in this example (an improved Transformer unit, excluding normalization layers, and excluding classification token features from input and output features) is... Figure 4 In this code, L is set to 12, and the encoder consists of 12 feature extraction layers. These 12 feature extraction layers are divided into 4 decoding modules. , , and The decoder consists of 12 feature reconstruction layers, which are divided into 4 decoding modules. , , and Each module consists of 3 layers.

[0193] In actual implementation, the encoder E Existing pre-trained models can be used; that is, the encoder can be a pre-trained encoder. During the training phase of the anomaly detection model, the model parameters (weights, etc.) of the pre-trained encoder are fixed. The encoder is used to map the input image to feature spaces of different depths, obtaining a cluster of feature maps, which are the first feature maps of multiple levels / depths. The encoding process of the encoder can be represented as follows:

[0194] (1)

[0195] in, i Represents the encoder's first i There are N+1 levels, where N+1 is the number of encoder levels, which is also the number of feature extraction modules. Figure 6 In the structure shown, N=4, and the feature extraction module includes an initial feature extraction module (not shown in the figure) and four encoding modules shown in the figure. i The feature map of the first stage (that is, the first...) i (+1 level of first feature map) ( F i-1 (that is) F i ) is the encoding module of the i-th stage used to encode the i-th... i -1 stage first feature map F i-1 For encoders with non-hierarchical columnar structures, the feature extraction is performed, and the first... i The resolution of the feature map in the first stage and the first stage i The feature maps in each of the -1 stages have the same resolution. For hierarchical encoders, such as... Figure 4 Structures (a) and (b) in the text, the first i The resolution of the feature map in the first stage and the first stage i The feature maps of the -1 stages have different resolutions. The feature map of the first stage is actually the first feature map F1 of the second level.

[0196] The feature fusion unit F, acting as a connecting component between the encoder and decoder, is capable of fusing feature information from multiple levels / stages of the encoder output. F i This results in a more compact fusion feature. The working principle of the fusion unit, fed into the decoder, can be represented as follows:

[0197] (2)

[0198] in, Used to adjust the i The coding features of the first stage will be...i The encoded features (feature maps) of each stage are adjusted to the required size for subsequent processing, such as upsampling or deconvolution operations, to ensure that the adjusted encoded features of each stage have the same resolution, allowing for the fusion of the adjusted encoded features from each stage. Specifically, if the encoding module uses a columnar rather than hierarchical encoding structure, the resolution of the encoded features at each stage is the same. The resolution of the encoded features in the i-th stage can be maintained without changing the resolution when only the encoded features of the last stage are used (e.g., ...). Figure 6 In F 4) When the anomaly detection model degenerates into a model with a structure similar to an autoencoder (AE), the anomaly detection model degenerates into a model similar to an autoencoder (AE).

[0199] decoder Use the fused features To reconstruct the original image or features, similar to an encoder, it contains N multi-stage feature reconstruction sub-models. That is, the decoding module. Figure 5 In the structure shown, N=4, and the decoding process of the decoding module can be represented as follows:

[0200] (3)

[0201] in, Indicates the relationship with the first i- The reconstructed feature map corresponding to the encoded features of one stage (first feature map). It is the decoding module of the i-th stage. Reconstructed features based on the encoded features of the i-th stage Obtained by feature reconstruction, such as Figure 6 As shown, the coding features of the third stage F 3. Reconstructed feature map It is a decoding module Based on fusion features The second-stage encoded feature map obtained through feature reconstruction. F 2 Corresponding Reconstructed Feature Map It is a decoding module Based on fusion features The features are obtained through feature reconstruction, and so on. The output is the same as F Reconstructed feature map corresponding to 0 .

[0202] Similarly, for a non-hierarchical columnar decoder, the first i The resolution of the reconstructed feature map corresponding to the first stage and the resolution of the second stage. i-The reconstructed feature maps for each stage have the same resolution. For hierarchical decoders, such as... Figure 5 The (a) and (c) structures in the text, the first i The resolution of the reconstructed feature map corresponding to the first stage and the resolution of the second stage. i- The resolution of the reconstructed feature maps varies for each stage.

[0203] Figure 7 A schematic diagram of an optional implementation process of the anomaly detection method provided in this application is shown, such as... Figure 7 As shown, in practice, this method can include a model training phase and a model application phase.

[0204] To better ensure the performance of the trained anomaly detection model, the model training phase provided in this application embodiment may include a model pre-training process ① and a model training process ② corresponding to the anomaly detection task, such as... Figure 6 As shown, the pre-training process ① is to train the encoder of the anomaly detection model to obtain a trained encoder that meets the first training termination condition. This encoder is then used as the encoder in the trained anomaly detection model. The training process ② is to train the other parts of the anomaly detection model (feature fusion unit and decoder) while keeping the model parameters of the pre-trained encoder fixed, to obtain an anomaly detection model that meets the second training termination condition.

[0205] It should be noted that, in practical applications, the specific training method in the pre-training stage is not limited to a single method in this application embodiment. As an optional solution, the pre-training weights of the encoder can be obtained by DINO (distillation with no labels, a self-supervised learning method), such as using ViT-Small pre-trained by DINO as the pre-training weights of the encoder.

[0206] The training process of the model can be executed by the training server 100. Optionally, after the training server 100 obtains an anomaly detection model that meets the second training termination condition by executing the above pre-training process ① and training process ②, it can also test the model using a test set. The test set can include multiple sample images known to have anomalies. The model is used to perform anomaly detection on these sample images. Based on the detection results and the actual situation of the sample images, it is determined whether the model's performance indicators meet the requirements (which can be set according to actual needs). If the requirements are met, the model can be used as a trained anomaly detection model. If the requirements are not met, the model can continue to be trained based on the training set until a model that meets the requirements is obtained.

[0207] The training image set during the training phase can include the first training set (classification sample set) required in the pre-training phase ① and the second training set (non-abnormal sample set) selected in the retraining phase ②. For example... Figure 6 As shown, the structure of the neural network model in the pre-training phase includes an encoder for the anomaly detection model to be trained and a classifier cascaded with the encoder. The classification sample set includes a large number of first sample images / training images with class labels. In the pre-training phase, the first sample images can be input into the neural network model. The encoder extracts sample feature maps of multiple levels of the first sample images. The classifier predicts the class of the sample images based on the sample feature maps of at least one level (including at least the last level). Since the true class of the first sample images is known, the training loss of the neural network model can be obtained based on the difference between the true class of each sample image and the class predicted by the model. If the first training termination condition is met (such as the number of training iterations reaching a set number or the model's loss function converging), the pre-training phase can be terminated. If the condition is not met, the model parameters of the encoder and classifier can be adjusted based on the training loss, and the above training process can be repeated.

[0208] The non-abnormal sample set contains a large number of normal sample images. Based on these normal sample images, the anomaly detection model can be further trained using an unsupervised training method. The neural network model in training phase ② is the entire anomaly detection model, including the pre-trained encoder, feature fusionist, and decoder, such as... Figure 6As indicated by the "lock" symbol, during training phase ②, the training optimizes the model parameters of the model part other than the encoder, while the model parameters of the pre-trained encoder remain fixed. The training process in the retraining phase includes: inputting normal sample images into the pre-trained encoder, extracting sample feature maps of multiple levels from the normal sample images through the encoder, fusing the sample feature maps of each level through a feature fusion processor, inputting the fused feature map into the decoder, and obtaining reconstructed sample feature maps corresponding to the sample feature maps of each level except the last level through the decoder. For each normal sample image, the sample anomaly map corresponding to each level can be obtained based on the pixel value differences between the sample feature maps of the M levels corresponding to the image (e.g., the sample feature maps of each level except the first level sample feature map F0) and the sample feature maps of the M levels and the corresponding reconstructed sample feature maps. The training loss corresponding to the sample image can be obtained based on the sample anomaly map of each level. During training, the non-abnormal sample set can be divided into multiple batches of images. Each training session uses one batch. The training loss of the model is obtained by calculating the sum or mean of the training losses corresponding to multiple normal sample images in a batch. The model parameters of the feature fusion unit and decoder are then adjusted based on this training loss until the second training termination condition is met, resulting in a well-trained anomaly detection model.

[0209] In the training phase ②, the model loss can be calculated based on multiple levels of anomaly maps corresponding to each sample image in a batch. A level's anomaly map is obtained based on the image difference (i.e., reconstruction error) between the anomaly map at that level and the reconstructed sample feature map corresponding to that level. Figure 6 As shown, A0 represents the first-level sample feature map and the corresponding reconstructed sample feature map, and A1 represents... Figure 6 The first encoding module outputs a sample feature map F1 and the corresponding reconstructed sample feature map shown. The corresponding sample anomaly plot, A2 represents F2 and The corresponding sample anomaly plot, A3 represents F3 and The corresponding sample anomaly map. Taking the i-th stage as an example, the sample anomaly map corresponding to this level... It can be calculated in the following ways:

[0210]

[0211] in, This represents the difference between the encoded feature map (i.e., the sample feature map) and the reconstructed sample feature map at the i-th stage. Optionally, the calculation method for this difference can include, but is not limited to, MAE, MSE, or cosine similarity. For non-hierarchical decoders, the resolution of the sample anomaly map, encoded feature map, and reconstructed sample feature map is the same. Specifically, the pixel value at each corresponding position in the sample anomaly map can be obtained by the difference between the pixel values ​​at each corresponding position in the encoded feature map and the reconstructed sample feature map.

[0212] Specifically, such as Figure 6 The coding features output by the four coding modules shown can be represented as follows: F 1 ~ F 4. In training phase ②, F 0 represents the initial feature map of the sample image, i.e., the first level of sample feature map. In the application phase... F 0 represents the initial feature map of the image to be detected, i.e., the first feature map of the first level. The fusion unit F can include at least... F Feature fusion is performed on feature maps at least one level, including 4. Optionally, the fusion unit F can be implemented using a single linear layer, the principle of which can be expressed as follows:

[0213]

[0214] Here, Conv1*1 represents a convolutional layer with a kernel size of 1*1. The fusion unit F concatenates feature maps from multiple levels of the encoder output. The concatenated features are then transformed by a 1*1 convolutional layer to obtain transformed features, i.e., the fused feature map, whose number equals the number of output channels of the encoding module. The fused feature map is input into the decoder, and the decoding module reconstructs the reconstructed feature map or reconstructed sample feature map based on the fused feature map. ~ Afterwards, formula (4) can be used to calculate the anomaly map or sample anomaly map corresponding to each level. Optionally, the sample anomaly map used to calculate the training loss during the training phase and the anomaly map used to determine the anomaly detection result of the image to be detected during the application phase may not use the anomaly map / sample anomaly map A0 corresponding to the first feature map / sample feature map of the first level, but only use the anomaly maps corresponding to some or all of the first feature map / sample feature map of other levels. For example, only anomaly map A3 can be used, or anomaly maps A1~A3 can be used.

[0215] In the application phase, taking the i-th phase as an example, assume the dimensions of the first feature map and the reconstructed feature map of the image to be detected are... The number of channels in the reconstructed feature map and the first feature map is Then, the first feature map and the reconstructed feature map of the i-th stage can be represented as follows: and , ( h,w () represents any location / pixel / feature point in the feature map, where the initial / original feature vector and reconstructed feature vector can be represented as: and The feature vector is... Ci Original feature map / reconstructed feature map of each channel ( h,w The vector is composed of the eigenvalues ​​at position (), and the dimension of the vector is (). Ci When using cosine similarity to calculate the differences between feature maps, anomaly maps... Middle position ( h,w Pixel value / outlier at ) It can be represented as follows:

[0216]

[0217] Using the above method, the anomaly map corresponding to each stage can be calculated. For each stage, the sum or mean of all pixel values ​​in the anomaly map corresponding to that stage can be used as the reconstruction loss / training loss for that level. For example, for the training stage, the loss function corresponding to the i-th stage can be expressed as follows:

[0218]

[0219] in, for The number of all positions / pixels in the data can be used to calculate the loss corresponding to the i-th stage using formula (7). The training loss corresponding to the sample image can be obtained by summing the reconstruction losses corresponding to each level. The sum or mean of the training losses corresponding to multiple sample images can be used as the training loss of the model to optimize the model parameters of the fusion and decoder.

[0220] After training a satisfactory anomaly detection model using training server 100, the model can be deployed to electronic devices that perform anomaly detection methods, such as... Figure 6 The anomaly detection device 200 in the model performs anomaly detection on the image to be detected and obtains the anomaly detection result. In the application stage, the input of the model is the image to be detected. The model can obtain the encoded feature map (first feature map) and reconstructed feature map of the image to be detected corresponding to each stage. The anomaly map of the image to be detected corresponding to each stage can be calculated by the above formulas (4) and (5). The anomaly maps corresponding to each stage can be fused to obtain the final anomaly map corresponding to the image to be detected. For example, the pixel values ​​of the corresponding positions in the anomaly maps corresponding to the first feature maps of each stage are added together to obtain the final anomaly map. The anomaly detection results of the image to be detected can be obtained from the final anomaly map, such as whether there are anomalies or where the anomaly regions are located.

[0221] Specifically, for anomaly localization, i.e., pixel-level anomaly detection, the locations in the final anomaly image A where the pixel value is greater than a preset pixel-level threshold can be identified as anomaly locations, and the image regions corresponding to each anomaly location are designated as anomaly regions. For sample-level anomaly detection, the final anomaly image can be processed using average pooling. Optionally, a pooling layer with a stride of 1 and a preset kernel size can be used to process the anomaly image. If the maximum pixel value in the processed anomaly image is greater than the preset pixel-level threshold, the image is determined to have an anomaly; otherwise, no anomaly is found.

[0222] The anomaly detection method provided in this application can use a model trained only on images without anomalies to identify abnormal images and locate abnormal regions. Therefore, it is well-suited for various practical application scenarios with anomaly detection requirements and can be widely used in, but not limited to, industrial defect detection, medical image detection, and video detection. For example, in industrial defect detection scenarios, this method can quickly and accurately detect whether generated industrial parts and other products have defects (i.e., anomalies), such as detecting whether transistors are qualified. The scheme provided in the optional embodiments of this application does not require a particularly complex network structure, and the amount of data computation can be effectively reduced, which can improve the efficiency of anomaly detection, especially in industrial anomaly detection scenarios with a large detection scale, where the effect is more obvious.

[0223] To verify the practicality of the solution provided in this application, we tested the anomaly detection model provided in the optional embodiments of this application and various anomaly detection models that use multi-scale feature maps on some public datasets (MVTec dataset, VisA dataset) on pixel-level anomaly detection tasks and pixel-level anomaly detection tasks. The model evaluation metrics used were AUROC, AUPRO and AP.

[0224] During testing, this application used an anomaly detection model based on the naive ViT-Small model, and tested this model along with several other models on test samples of various categories (including bottles, cables, pills, screws, carpets, etc.). Specifically, on the MVTec dataset, the proposed solution achieved AUROC / AP scores of 98.1 / 99.3 at the sample level and AUROC / AUPRO / AP scores of 97.5 / 90.7 / 53.2 at the pixel level. On the VisA dataset, the proposed solution achieved AUROC / AP scores of 94.0 / 92.2 at the sample level and AUROC / AUPRO / AP scores of 81.1 / 98.2 / 37.6 at the pixel level, achieving the best test results compared to other proposed solutions.

[0225] As an example, Figure 8 This diagram illustrates the visualized detection effects of various schemes on test samples. The "Normal" column represents normal sample images, the "Abnormal" column represents abnormal sample images corresponding to the normal sample images, and the "True Value" column corresponds to the true results of abnormal regions in each abnormal sample image (white areas in the diagram represent abnormal regions). The last column represents the schemes of this application embodiment. Schemes 1 to 5 are various schemes based on other network structures for anomaly detection models. Each scheme's corresponding column of images is a visual representation of the scheme's detection results for each abnormal sample image (due to the influence of image display effects in text, the abnormal region detection results are not particularly clear, but this does not affect the explanation). The area enclosed by the white curve in the detection effect image represents the detected abnormal region. Comparing the true value images of the abnormal sample images with the detection results of various schemes shows that the scheme of this application embodiment has a more accurate and compact localization capability for abnormal regions and a more stable low-anomaly detection result for normal regions, meaning that the probability of detecting normal regions as abnormal regions is lower. Therefore, the schemes provided by this application embodiment have good practical value, and the detection effect can well meet application requirements.

[0226] Based on the same principles as the methods provided in the embodiments of this application, the embodiments of this application also provide an anomaly detection device based on artificial intelligence, such as... Figure 9 As shown, the anomaly detection device 1000 includes a target image acquisition module 110, a feature extraction and reconstruction module 120, and an anomaly determination module 130, wherein:

[0227] The image acquisition module 110 is used to acquire the image to be detected;

[0228] The feature extraction and reconstruction module 120 is used to extract features from the image to be detected using a trained anomaly detection model, to obtain first feature maps at multiple levels, and to reconstruct the first feature maps of at least some of the multiple levels based on intermediate feature maps, to obtain reconstructed feature maps corresponding to the first feature maps of each of the at least some levels, wherein the intermediate feature maps belong to the first feature maps of the multiple levels and include at least the first feature map of the last level among the multiple levels, and the at least some levels do not include the last level;

[0229] The anomaly determination module 130 is used to obtain anomaly maps corresponding to each of the M levels based on the first feature maps of the M levels and the reconstructed feature maps corresponding to the first feature maps of the M levels, and to identify the anomaly detection results of the image to be detected based on the anomaly maps corresponding to the M levels, wherein M is a positive integer, anomaly map corresponding to a level is used to characterize the pixel difference between the first feature map and the reconstructed feature map of the level, and the M levels belong to the at least some levels;

[0230] The trained anomaly detection model is obtained by training the anomaly detection model to be trained based on sample images without anomalies and training loss. The training loss is used to characterize the difference between the M-level sample feature maps obtained by feature extraction of the sample images and the reconstructed sample feature maps corresponding to the M-level sample features.

[0231] Optionally, the resolution of the first feature map and the reconstructed feature map of each of the above multiple levels is the same.

[0232] Optionally, the anomaly determination module can be used to: fuse the anomaly maps corresponding to M levels to obtain the final anomaly map, and identify the anomaly detection result of the image to be detected based on the final anomaly map.

[0233] Optionally, the anomaly determination module can be used to: obtain the pixel value of the corresponding position in the final anomaly map by adding the pixel values ​​of the same position in the anomaly maps corresponding to M levels.

[0234] Optionally, the anomaly detection module can identify the anomaly detection results of the image to be detected by performing at least one of the following:

[0235] The final anomaly map is divided into regions and pixel values ​​are fused. The fused result of each pixel value in each region is used as the anomaly score of that region. Based on the anomaly score of each region, it is determined whether there is anomaly in the image to be detected. The size of each region is a set size.

[0236] The pixel value at each location in the final anomaly image is used as the anomaly score for that location. If there are locations in the final anomaly image with an anomaly score not less than the second threshold, the image regions corresponding to the locations in the image to be detected with an anomaly score not less than the second threshold in the final anomaly image are determined as anomaly regions.

[0237] Optionally, both the first feature map and the reconstructed feature map of each level have multiple channels, and the number of channels in the first feature map and the number of channels in the reconstructed feature map of a level are the same; for each of the M levels, the anomaly map corresponding to that level is obtained in the following way:

[0238] For each identical position in the first feature map and the reconstructed feature map of this level, the pixel values ​​of that position in multiple channels of the first feature map (that is, the pixel values ​​of that position in the feature maps of multiple channels) are combined to obtain the original feature vector corresponding to that position, and the pixel values ​​of that position in multiple channels of the reconstructed feature map are combined to obtain the reconstructed feature vector corresponding to that position.

[0239] For each identical location, determine the degree of difference between the original feature vector and the reconstructed feature vector corresponding to that location, and use this degree of difference as the pixel value of that location in the anomaly map corresponding to that level.

[0240] Optionally, the image acquisition module is used to: determine the anomaly detection task type, which is either a first type or a second type. The first type is used to detect whether there is an anomaly in the image, and the second type is used to detect abnormal regions in the image; when the task type is the first type, acquire the image to be detected with a resolution of a first set resolution; when the task type is the second type, acquire the image to be detected with a resolution of a second set resolution, wherein the second set resolution is greater than the first set resolution.

[0241] Optionally, at least some of the levels include all of the multiple levels except the last one, and the M levels include all of the multiple levels except the first one.

[0242] Optionally, the trained anomaly detection model includes an encoder and a decoder, wherein the encoder includes multiple cascaded encoding modules and the decoder includes multiple cascaded decoding modules.

[0243] The first feature maps of multiple levels, and the reconstructed feature maps corresponding to the first feature maps of at least some of the levels, are obtained in the following way:

[0244] The image to be detected is input into the encoder, and features are extracted sequentially through each encoding module of the encoder to obtain the first feature map of a level corresponding to each encoding module. The input of the next encoding module is the first feature map output by the previous encoding module.

[0245] Based on the intermediate feature map, the first feature map of each level in at least some levels is reconstructed by each decoding module in the decoder, so as to obtain the reconstructed feature map corresponding to the first feature map of each level in at least some levels.

[0246] Optionally, the anomaly detection model also includes a feature fusion unit that connects the encoder and decoder, wherein the reconstructed feature map corresponding to the first feature map of each of at least some of the above layers is obtained in the following way:

[0247] The first feature maps of each level in the intermediate feature maps are fused using a feature fusion tool to obtain a fused feature map.

[0248] The fused feature map is input into the decoder, and the reconstructed feature map corresponding to the first feature map of each level is obtained through the reconstruction of each decoding module.

[0249] Optionally, the neural network structures of each encoding module and each decoding module are the same.

[0250] Optionally, the encoder and decoder are based on a neural network structure of the Naive Vision Transformer (ViT) network, where each encoding module includes at least one cascaded feature extraction layer, and each decoding module includes at least one cascaded feature reconstruction layer. The neural network structure satisfies at least one of the following:

[0251] Each feature extraction layer and each feature reconstruction layer correspond to a Transformer encoding module in the ViT network. The Transformer encoding module does not include a normalization layer.

[0252] The inputs and outputs of the Transformer encoding module do not include classification token features;

[0253] The feature fusion unit includes at least one linear layer.

[0254] Optionally, the above feature extraction and reconstruction module can be used for:

[0255] The image to be detected is divided into multiple image blocks according to the target size;

[0256] For each image block, the image feature vector of the image block is obtained based on the pixel information of the image block, the position feature vector of the image block is obtained according to the position of the image block in the image to be detected, and the initial feature vector of the image block is obtained by fusing the image feature vector and the position feature vector of the image block;

[0257] The initial feature vectors of multiple image patches are used as input to the encoder, and features are extracted sequentially by each encoding module in the encoder.

[0258] Optionally, the above fused feature map is obtained in the following way:

[0259] The first feature maps of each level in the intermediate feature maps are concatenated by a feature fusion processor along the channel dimension, and the concatenated feature maps are then transformed to obtain a fused feature map with the number of channels equal to the number of output channels of the encoding module.

[0260] Optionally, the feature extraction and reconstruction module is also used for:

[0261] Determine the anomaly detection task type, which is either type 1 or type 2. Type 1 is used to detect whether there are anomalies in the image, while type 2 is used to detect abnormal regions in the image.

[0262] If the task type is type 1, the first dimension is determined as the target dimension; if the task type is type 2, the second dimension is determined as the target dimension, and the second dimension is smaller than the first dimension.

[0263] Optionally, the trained anomaly detection model is obtained by training in the following way:

[0264] Obtain the first training set, which includes multiple labeled training images. The label of a training image represents the true category of that training image.

[0265] The first neural network model is trained on the first training set until the first training termination condition is met. The first neural network model includes an encoder of the anomaly detection model to be trained and a classifier cascaded with the encoder. The input and output of the first neural network model are the training image and the predicted category of the training image, respectively. The training loss of the first neural network model represents the difference between the true category and the predicted category of each training image.

[0266] Obtain a second training set, which includes multiple non-abnormal sample images (i.e., sample images without abnormalities), wherein the multiple sample images include images of sample objects of multiple categories;

[0267] The encoder that meets the first training termination condition is taken as the trained encoder. The model parameters of the trained encoder are fixed. The part of the anomaly detection model to be trained, excluding the encoder, is trained based on the second training set until the second training termination condition is met, and the trained anomaly detection model is obtained.

[0268] Optionally, the anomaly detection model to be trained can be repeatedly trained based on the second training set until the second training termination condition is met, thereby obtaining a trained anomaly detection model. The training operation includes:

[0269] Each of the sample images is input into the anomaly detection model to be trained, resulting in M-level sample feature maps and reconstructed sample feature maps corresponding to each sample image. Based on the differences between the M-level sample feature maps and the reconstructed sample feature maps, a training loss is obtained. If the second training termination condition is not met, the model parameters of the model part other than the encoder in the anomaly detection model to be trained are adjusted based on the training loss.

[0270] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0271] This application also provides an electronic device including at least one processor configured to execute the steps of the method provided in any optional embodiment of this application. Optionally, the electronic device may further include a transceiver and / or a memory coupled to the processor, the memory storing a computer program, which the processor can use to implement the scheme provided in any optional embodiment of this application when running the computer program. Optionally, the electronic device may be a user terminal or a server.

[0272] Figure 10 A schematic diagram of the structure of an electronic device to which an embodiment of the present invention applies is shown, such as... Figure 10 As shown, the electronic device can be a server or a user terminal, and it can be used to implement the methods provided in any embodiment of the present invention.

[0273] like Figure 10 As shown, the electronic device 2000 may primarily include at least one processor 2001. Figure 10 The diagram shows components such as a memory 2002, a communication module 2003, and an input / output interface 2004. Optionally, these components can be connected and communicate with each other via a bus 2005. It should be noted that... Figure 10 The structure of the electronic device 2000 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable.

[0274] The memory 2002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0275] Processor 2001 is connected to memory 2002 via bus 2005, and implements corresponding functions by calling application programs stored in memory 2002. Processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0276] Electronic device 2000 can connect to a network via communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 2003 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules.

[0277] Electronic device 2000 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 2004. Electronic device 2000 itself may have a display device, and other display devices can also be connected externally via interface 2004. Optionally, storage devices, such as hard drives, can also be connected via interface 2004 to store data from electronic device 2000, retrieve data from storage devices, or store data from storage devices into memory 2002. It is understood that input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 2004 can be a component of electronic device 2000 or an external device connected to electronic device 2000 when needed.

[0278] The bus 2005 used to connect the components may include a pathway for transmitting information between the components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 2005 can be divided into address bus, data bus, control bus, etc.

[0279] Optionally, for the solution provided in the embodiments of the present invention, the memory 2002 can be used to store a computer program that executes the solution of the present invention, and the processor 2001 runs the computer program. When the processor 2001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of the present invention.

[0280] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0281] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0282] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0283] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0284] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. An anomaly detection method based on artificial intelligence, characterized in that, The method includes: Acquire the image to be detected; The trained anomaly detection model is used to extract features from the image to be detected, resulting in first feature maps of multiple levels. Based on the intermediate feature maps, at least some of the first feature maps of the multiple levels are reconstructed to obtain reconstructed feature maps corresponding to the first feature maps of each of the at least some levels. The intermediate feature maps belong to the first feature maps of the multiple levels and include at least the first feature map of the last level of the multiple levels. The resolution of the first feature maps and the reconstructed feature maps of each level is the same. Based on the first feature maps of M levels and the reconstructed feature maps corresponding to the first feature maps of the M levels, anomaly maps corresponding to each of the M levels are obtained, where M is a positive integer. An anomaly map corresponding to a level is used to characterize the pixel difference between the first feature map and the reconstructed feature map of that level. The M levels belong to the at least some levels. The anomaly detection results of the image to be detected are identified based on the anomaly maps corresponding to the M levels; The trained anomaly detection model is obtained by training the anomaly detection model to be trained based on non-abnormal sample images and training loss. The training loss is used to characterize the difference between the M-level sample feature maps obtained by feature extraction of the sample images and the reconstructed sample feature maps corresponding to the M-level sample feature maps.

2. The method according to claim 1, characterized in that, The anomaly detection result of the image to be detected based on the anomaly maps corresponding to the M levels includes: The anomaly graphs corresponding to the M levels are fused to obtain the final anomaly graph; The anomaly detection result of the image to be detected is identified based on the final anomaly map.

3. The method according to claim 2, characterized in that, The process of fusing the anomaly graphs corresponding to the M levels to obtain the final anomaly graph includes: By adding the pixel values ​​at the same position in the anomaly maps corresponding to the M levels, the pixel value at the corresponding position in the final anomaly map is obtained. The anomaly detection result of the image to be detected based on the final anomaly map includes at least one of the following: The final abnormal image is divided into regions and pixel values ​​are fused. The fused result of each pixel value in each region is used as the abnormal score of that region. Based on the abnormal scores of each region, it is identified whether there is an abnormality in the image to be detected. The size of each region is a set size. The pixel value of each position in the final abnormal image is used as the abnormal score for that position. If there is a position in the final abnormal image with an abnormal score not less than the second threshold, the image region corresponding to each position in the final abnormal image with an abnormal score not less than the second threshold in the image to be detected is determined as an abnormal region.

4. The method according to claim 1, characterized in that, The first feature map and the reconstructed feature map of each level have multiple channels, and the number of channels in the first feature map and the number of channels in the reconstructed feature map of a level are the same. The anomaly map corresponding to each of the M levels is obtained by using the first feature map based on the M levels and the reconstructed feature map corresponding to the first feature map of the M levels, including: For each of the M levels, for each identical position in the first feature map and the reconstructed feature map of that level, the pixel values ​​of that position in multiple channels of the first feature map are combined to obtain the original feature vector corresponding to that position, and the pixel values ​​of that position in multiple channels of the reconstructed feature map are combined to obtain the reconstructed feature vector corresponding to that position. The degree of difference between the original feature vector and the reconstructed feature vector corresponding to that position is determined, and the degree of difference is used as the pixel value of that position in the anomaly map corresponding to that level.

5. The method according to claim 1, characterized in that, The acquisition of the image to be detected includes: Determine the anomaly detection task type, which is either a first type or a second type. The first type is used to detect whether there are anomalies in the image, and the second type is used to detect abnormal regions in the image. When the task type is the first type, an image to be detected with a resolution of the first set resolution is acquired; When the task type is the second type, an image to be detected with a resolution of a second set resolution is acquired, wherein the second set resolution is greater than the first set resolution.

6. The method according to claim 1, characterized in that, The at least partial level includes each of the plurality of levels except the last level, and the M levels include each of the at least partial level except the first level of the plurality of levels.

7. The method according to any one of claims 1 to 6, characterized in that, The trained anomaly detection model includes an encoder and a decoder. The encoder includes multiple cascaded encoding modules, and the decoder includes multiple cascaded decoding modules. The first feature maps of the multiple layers, and the reconstructed feature maps corresponding to the first feature maps of each of the at least some layers, are obtained in the following manner: The image to be detected is input into the encoder, and features are extracted sequentially by each encoding module of the encoder to obtain a first feature map of a level corresponding to each encoding module. The input of the next encoding module is the first feature map output by the previous encoding module. Based on the intermediate feature map, the first feature map of each level in the at least some levels is reconstructed by each decoding module in the decoder to obtain the reconstructed feature map corresponding to the first feature map of each level in the at least some levels.

8. The method according to claim 7, characterized in that, The anomaly detection model also includes a feature fusion unit that connects the encoder and the decoder; The reconstructed feature map corresponding to the first feature map of each of at least some of the layers is obtained in the following way: The feature fusion processor performs feature fusion on the first feature maps of each level in the intermediate feature map to obtain a fused feature map. The fused feature map is input into the decoder, and the reconstructed feature map corresponding to the first feature map of each of the at least some layers is obtained by each of the decoding modules.

9. The method according to claim 8, characterized in that, The encoder and the decoder are based on a neural network structure of Naive Visual Transformation (ViT) network. Each encoding module includes at least one cascaded feature extraction layer, and each decoding module includes at least one cascaded feature reconstruction layer. The neural network structure satisfies at least one of the following: Each feature extraction layer and each feature reconstruction layer correspond to a Transformer encoding module in the ViT network, and the Transformer encoding module does not include a normalization layer; The inputs and outputs of the Transformer encoding module do not include classification token features; The feature fusion unit includes at least one linear layer; The step of inputting the image to be detected into the encoder and extracting features sequentially through each encoding module of the encoder includes: The image to be detected is divided into multiple image blocks according to the target size; For each image block, an image feature vector is obtained based on the pixel information of the image block, and a position feature vector is obtained based on the position of the image block in the image to be detected. An initial feature vector of the image block is obtained by fusing the image feature vector and the position feature vector. The initial feature vectors of the multiple image patches are used as input to the encoder, and feature extraction is performed sequentially by each encoding module in the encoder. The step of fusing the first feature maps of each level in the intermediate feature map using the feature fusion processor to obtain a fused feature map includes: The feature fusion processor concatenates the first feature maps of each level in the intermediate feature map along the channel dimension, and performs feature transformation on the concatenated feature map to obtain a fused feature map with the number of channels equal to the number of output channels of the encoding module.

10. The method according to claim 9, characterized in that, The method further includes: Determine the anomaly detection task type, which is either a first type or a second type. The first type is used to detect whether there is an anomaly in the image, and the second type is used to detect abnormal regions in the image. If the task type is a first type, the first size is determined as the target size; if the task type is a second type, the second size is determined as the target size, and the second size is smaller than the first size.

11. The method according to claim 7, characterized in that, The trained anomaly detection model was obtained through the following method: Obtain a first training set, which includes multiple labeled training images, where the label of a training image represents the true category of that training image. The first neural network model is trained based on the first training set until the first training termination condition is met. The first neural network model includes an encoder of the anomaly detection model to be trained and a classifier cascaded with the encoder. The input and output of the first neural network model are the training image and the predicted category of the training image, respectively. The training loss of the first neural network model represents the difference between the true category and the predicted category of each training image. Obtain a second training set, which includes multiple non-abnormal sample images, wherein the multiple sample images include images of sample objects of multiple categories; The encoder that meets the first training termination condition is taken as the trained encoder. The training operation is repeated on the anomaly detection model to be trained based on the second training set until the second training termination condition is met, thus obtaining the trained anomaly detection model. The training operation includes: Each of the sample images is input into the anomaly detection model to be trained, resulting in M-level sample feature maps and reconstructed sample feature maps corresponding to each sample image. Based on the differences between the M-level sample feature maps and the reconstructed sample feature maps, a training loss is obtained. If the second training termination condition is not met, the model parameters of the model part other than the encoder in the anomaly detection model to be trained are adjusted based on the training loss.

12. An anomaly detection device based on artificial intelligence, characterized in that, The device includes: The image acquisition module is used to acquire the image to be detected. The feature extraction and reconstruction module is used to extract features from the image to be detected using a trained anomaly detection model to obtain first feature maps at multiple levels, and to reconstruct the first feature maps at least some of the multiple levels based on intermediate feature maps to obtain reconstructed feature maps corresponding to the first feature maps at each of the at least some levels. The intermediate feature maps belong to the first feature maps of the multiple levels and include at least the first feature map of the last level among the multiple levels. The resolution of the first feature maps and the reconstructed feature maps at each level is the same. An anomaly determination module is used to obtain an anomaly map corresponding to each of the M levels based on the first feature maps of the M levels and the reconstructed feature maps corresponding to the first feature maps of the M levels, and to identify the anomaly detection result of the image to be detected based on the anomaly maps corresponding to the M levels, wherein M is a positive integer, an anomaly map corresponding to a level is used to characterize the pixel difference between the first feature map and the reconstructed feature map of the level, and the M levels belong to the at least some levels; The trained anomaly detection model is obtained by training the anomaly detection model to be trained based on non-abnormal sample images and training loss. The training loss is used to characterize the difference between the M-level sample feature maps obtained by feature extraction of the sample images and the reconstructed sample feature maps corresponding to the M-level sample feature maps.

13. The apparatus according to claim 12, characterized in that, When the anomaly determination module identifies the anomaly detection results of the image to be detected based on the anomaly maps corresponding to the M levels, it is specifically used for: The anomaly graphs corresponding to the M levels are fused to obtain the final anomaly graph; The anomaly detection result of the image to be detected is identified based on the final anomaly map.

14. The apparatus according to claim 13, characterized in that, When the anomaly determination module fuses the anomaly graphs corresponding to the M levels to obtain the final anomaly graph, it is specifically used for: By adding the pixel values ​​at the same position in the anomaly maps corresponding to the M levels, the pixel value at the corresponding position in the final anomaly map is obtained. The anomaly detection module identifies the anomaly detection result of the image to be detected by performing at least one of the following: The final abnormal image is divided into regions and pixel values ​​are fused. The fused result of each pixel value in each region is used as the abnormal score of that region. Based on the abnormal scores of each region, it is identified whether there is an abnormality in the image to be detected. The size of each region is a set size. The pixel value of each position in the final abnormal image is used as the abnormal score for that position. If there is a position in the final abnormal image with an abnormal score not less than the second threshold, the image region corresponding to each position in the final abnormal image with an abnormal score not less than the second threshold in the image to be detected is determined as an abnormal region.

15. The apparatus according to claim 12, characterized in that, The first feature map and the reconstructed feature map of each level have multiple channels, and the number of channels in the first feature map and the number of channels in the reconstructed feature map of a level are the same. For each of the M levels, the anomaly graph corresponding to that level is obtained in the following way: For each of the M levels, for each identical position in the first feature map and the reconstructed feature map of that level, the pixel values ​​of that position in multiple channels of the first feature map are combined to obtain the original feature vector corresponding to that position, and the pixel values ​​of that position in multiple channels of the reconstructed feature map are combined to obtain the reconstructed feature vector corresponding to that position. The degree of difference between the original feature vector and the reconstructed feature vector corresponding to that position is determined, and the degree of difference is used as the pixel value of that position in the anomaly map corresponding to that level.

16. The apparatus according to claim 12, characterized in that, The image acquisition module is specifically used to acquire the image to be detected when: Determine the anomaly detection task type, which is either a first type or a second type. The first type is used to detect whether there are anomalies in the image, and the second type is used to detect abnormal regions in the image. When the task type is the first type, an image to be detected with a resolution of the first set resolution is acquired; When the task type is the second type, an image to be detected with a resolution of a second set resolution is acquired, wherein the second set resolution is greater than the first set resolution.

17. The apparatus according to claim 12, characterized in that, The at least partial level includes each of the plurality of levels except the last level, and the M levels include each of the at least partial level except the first level of the plurality of levels.

18. The apparatus according to any one of claims 12-17, characterized in that, The trained anomaly detection model includes an encoder and a decoder. The encoder includes multiple cascaded encoding modules, and the decoder includes multiple cascaded decoding modules. The first feature maps of the multiple layers, and the reconstructed feature maps corresponding to the first feature maps of each of the at least some layers, are obtained in the following manner: The image to be detected is input into the encoder, and features are extracted sequentially by each encoding module of the encoder to obtain a first feature map of a level corresponding to each encoding module. The input of the next encoding module is the first feature map output by the previous encoding module. Based on the intermediate feature map, the first feature map of each level in the at least some levels is reconstructed by each decoding module in the decoder to obtain the reconstructed feature map corresponding to the first feature map of each level in the at least some levels.

19. The apparatus according to claim 18, characterized in that, The anomaly detection model also includes a feature fusion unit that connects the encoder and the decoder; The reconstructed feature map corresponding to the first feature map of each of at least some of the layers is obtained in the following way: The feature fusion processor performs feature fusion on the first feature maps of each level in the intermediate feature map to obtain a fused feature map. The fused feature map is input into the decoder, and the reconstructed feature map corresponding to the first feature map of each of the at least some layers is obtained by each of the decoding modules.

20. The apparatus according to claim 19, characterized in that, The encoder and the decoder are based on a neural network structure of Naive Visual Transformation (ViT) network. Each encoding module includes at least one cascaded feature extraction layer, and each decoding module includes at least one cascaded feature reconstruction layer. The neural network structure satisfies at least one of the following: Each feature extraction layer and each feature reconstruction layer correspond to a Transformer encoding module in the ViT network, and the Transformer encoding module does not include a normalization layer; The inputs and outputs of the Transformer encoding module do not include classification token features; The feature fusion unit includes at least one linear layer; The feature extraction and reconstruction module is used for: The image to be detected is divided into multiple image blocks according to the target size; For each image block, an image feature vector is obtained based on the pixel information of the image block, and a position feature vector is obtained based on the position of the image block in the image to be detected. An initial feature vector of the image block is obtained by fusing the image feature vector and the position feature vector. The initial feature vectors of the multiple image patches are used as input to the encoder, and feature extraction is performed sequentially by each encoding module in the encoder. The fused feature map is obtained through the following method: The feature fusion processor concatenates the first feature maps of each level in the intermediate feature map along the channel dimension, and performs feature transformation on the concatenated feature map to obtain a fused feature map with the number of channels equal to the number of output channels of the encoding module.

21. The apparatus according to claim 20, characterized in that, The feature extraction and reconstruction module is also used for: Determine the anomaly detection task type, which is either a first type or a second type. The first type is used to detect whether there is anomaly detection in the image, and the second type is used to detect abnormal regions in the image. If the task type is a first type, the first size is determined as the target size; if the task type is a second type, the second size is determined as the target size, and the second size is smaller than the first size.

22. The apparatus according to claim 18, characterized in that, The trained anomaly detection model was obtained through the following method: Obtain a first training set, which includes multiple labeled training images, where the label of a training image represents the true category of that training image. The first neural network model is trained based on the first training set until the first training termination condition is met. The first neural network model includes an encoder of the anomaly detection model to be trained and a classifier cascaded with the encoder. The input and output of the first neural network model are the training image and the predicted category of the training image, respectively. The training loss of the first neural network model represents the difference between the true category and the predicted category of each training image. Obtain a second training set, which includes multiple non-abnormal sample images, wherein the multiple sample images include images of sample objects of multiple categories; The encoder that meets the first training termination condition is taken as the trained encoder. The training operation is repeated on the anomaly detection model to be trained based on the second training set until the second training termination condition is met, thus obtaining the trained anomaly detection model. The training operation includes: Each of the sample images is input into the anomaly detection model to be trained, resulting in M-level sample feature maps and reconstructed sample feature maps corresponding to each sample image. Based on the differences between the M-level sample feature maps and the reconstructed sample feature maps, a training loss is obtained. If the second training termination condition is not met, the model parameters of the model part other than the encoder in the anomaly detection model to be trained are adjusted based on the training loss.

23. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the method according to any one of claims 1 to 11 when running the computer program.

24. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images