Abnormality detection method, model training mode, device, medium and program product

Abstract

The invention provides an anomaly detection method, a model training mode and device, a medium and a program product, and relates to the AI technology, the anomaly detection method comprises the steps: carrying out the multi-scale feature processing of a first image through a feature extraction network, and obtaining a first multi-scale feature sequence; reconstructing the input feature sequence of the layer based on the reference feature sequence of the layer through the attention network in the feature reconstruction network of the ith layer to obtain a first reconstructed feature sequence; processing the first reconstructed feature sequence through a summation and normalization layer in the ith layer of feature reconstruction network to obtain a second reconstructed feature sequence; processing the second reconstruction feature sequence through other network layers in the ith layer feature reconstruction network to obtain a third reconstruction feature sequence; based on the first multi-scale feature sequence and the final third reconstruction feature sequence, determining a feature difference score graph of the first image; and determining abnormal pixel points based on the score map. Therefore, the reliability of anomaly detection can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence (AI) technology, and particularly relate to an anomaly detection method, a model training manner, an apparatus, a medium and a program product. BACKGROUND

[0002] Anomaly detection technology is an important technology in industrial AI quality inspection, and the purpose is to find abnormal defects on industrial parts.

[0003] The current anomaly detection method includes: an electronic device performs multi-scale feature processing on a to-be-detected image to obtain a multi-scale feature sequence; and reconstructs the multi-scale feature sequence through a feature reconstruction network to obtain a reconstructed feature sequence; and finally performs anomaly detection based on the multi-scale feature sequence and the reconstructed feature sequence.

[0004] Among them, the current feature reconstruction network is a Transfomer structure, and this structure has a learning shortcut problem, in other words, this structure is not easy to learn the normal image feature mode, so that for the actual abnormal image, it is not easy to reconstruct the feature sequence conforming to the normal image feature mode through the feature reconstruction network of this structure, and further leads to the problem of low reliability of anomaly detection based on the difference between the multi-scale feature sequence and the reconstructed feature sequence. SUMMARY

[0005] The present application provides an anomaly detection method, a model training manner, an apparatus, a medium and a program product, thereby improving the reliability of anomaly detection.

[0006] In a first aspect, an embodiment of the present application provides an anomaly detection method, comprising: performing multi-scale feature processing on a first image by a feature extraction network to obtain a first multi-scale feature sequence; reconstructing, by each of M attention networks in an i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence in the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the first multi-scale feature sequence, i=1, 2, …, N, N and M are positive integers, and the reference feature sequence in the i-th layer feature reconstruction network conforms to a normal image feature mode; if M is equal to 1, performing normalization processing on the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, performing weighted summation on the M first reconstructed feature sequences by the sum and normalization layer, and performing normalization processing on a weighted summation result to obtain the second reconstructed feature sequence; processing the second reconstructed feature sequence by other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence; determining a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by an N-th layer feature reconstruction network; and determining an abnormal pixel point of the first image based on the feature difference score map of the first image.

[0007] In a second aspect, an embodiment of the present application provides a model training method, comprising: performing multi-scale feature processing on a second image by a feature extraction network to obtain a second multi-scale feature sequence; reconstructing, by each of M attention networks in an i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence in the i-th layer feature reconstruction network to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the second multi-scale feature sequence, i=1, 2, …, N, N and M are positive integers; if M is equal to 1, performing normalization processing on the fourth reconstructed feature sequence by a sum and normalization layer to obtain a fifth reconstructed feature sequence; or if M is greater than 1, performing weighted summation on the M fourth reconstructed feature sequences by the sum and normalization layer, and performing normalization processing on a weighted summation result to obtain the fifth reconstructed feature sequence; processing the fifth reconstructed feature sequence by other network layers in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence; calculating a loss of a target model composed of the feature extraction network and N layer feature reconstruction networks based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th layer feature reconstruction network; and training the target model and the reference feature sequence of each layer in the N layer feature reconstruction network based on the loss of the target model.

[0008] In a third aspect, an embodiment of the present application provides an anomaly detection device, comprising: a multi-scale feature processing module, configured to perform multi-scale feature processing on a first image by a feature extraction network to obtain a first multi-scale feature sequence;

[0009] A first feature reconstruction module is configured to reconstruct an input feature sequence of an i-th layer feature reconstruction network based on a reference feature sequence in the i-th layer feature reconstruction network by each attention network in the M attention networks in the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the first multi-scale feature sequence, i=1, 2, …, N, N and M are positive integers, and the reference feature sequence in the i-th layer feature reconstruction network conforms to a normal image feature mode;

[0010] A second feature reconstruction module is configured to perform normalization processing on the first reconstructed feature sequence by a summation and normalization layer to obtain a second reconstructed feature sequence if M is equal to 1, or to perform weighted summation on the M first reconstructed feature sequences by the summation and normalization layer and perform normalization processing on a weighted summation result to obtain the second reconstructed feature sequence if M is greater than 1;

[0011] A third feature reconstruction module is configured to process the second reconstructed feature sequence by other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence;

[0012] A first determination module is configured to determine a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th layer feature reconstruction network;

[0013] A second determination module is configured to determine an abnormal pixel point of the first image based on the feature difference score map of the first image.

[0014] In some implementable manners, the M attention networks comprise at least one of the following: a first attention network and a second attention network; correspondingly, the first feature reconstruction module is configured to perform at least one of the following:

[0015] The first feature reconstruction module is configured to perform at least one of the following:

[0016] The second attention network is used for converting each feature in the input feature sequence in the i-th layer feature reconstruction network into a second query vector, and converting each feature in the reference feature sequence in the i-th layer feature reconstruction network into a second key vector and a second value vector; a second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the second query vector and the second key vector; and a first reconstructed feature sequence is calculated based on the second attention matrix and the second value vector.

[0017] In some implementations, the first feature reconstruction module is specifically configured to: perform scaled dot product between the first query vector corresponding to the first feature and the transpose of each first key vector to obtain a third attention matrix corresponding to the first feature; and calculate a sum of the third attention matrix corresponding to the first feature and a first mask matrix corresponding to the first feature, and perform normalization processing to obtain the first attention matrix corresponding to the first feature; wherein the values of the first feature corresponding position and the surrounding positions in the first mask matrix are negative infinity, and the values of the remaining positions are 0.

[0018] In some implementations, the first feature reconstruction module is specifically configured to: perform scaled dot product between the second query vector corresponding to the second feature and the transpose of each second key vector to obtain a fourth attention matrix corresponding to the second feature; wherein the second feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; calculate a sum of the fourth attention matrix corresponding to the second feature and a second mask matrix corresponding to the second feature, and perform normalization processing to obtain a second attention matrix corresponding to the second feature; wherein the values of the second feature corresponding position and the surrounding positions in the second mask matrix are 0, and the values of the remaining positions are negative infinity.

[0019] In some implementations, when the first reconstructed feature sequence output by the first attention network and the first reconstructed feature sequence output by the second attention network are weighted and summed by the sum and normalization layer, the weight corresponding to the first reconstructed feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstructed feature sequence output by the second attention network is a hyperparameter greater than 1.

[0020] In some implementations, the multi-scale feature processing module is specifically configured to: perform multi-scale feature extraction on the first image by a pre-trained model in the feature extraction network to obtain a first feature map under multiple scales; perform fusion on the first feature map under multiple scales by a fusion layer in the feature extraction network to obtain a first fused feature map; and perform shape conversion on the first fused feature map by a conversion layer in the feature extraction network to obtain a first multi-scale feature sequence.

[0021] In some implementations, the multi-scale feature processing module is specifically used to: perform average pooling operations on the first feature maps at multiple scales through a fusion layer to obtain multiple second feature maps at the same scale; and merge the multiple second feature maps at the same scale along the channel dimension through a fusion layer to obtain a first fused feature map.

[0022] In some possible implementations, the first determining module is specifically used to: restore the third reconstructed feature sequence output by the Nth layer feature reconstruction network to a third feature map at multiple scales; restore the first multi-scale feature sequence to a fourth feature map at multiple scales; and calculate the feature difference score between the third feature map at multiple scales and the fourth feature map at multiple scales to obtain the feature difference score map of the first image.

[0023] In some implementations, the second determining module is specifically used to: determine pixels in the feature difference score map that have a score higher than a preset score as abnormal pixels in the first image.

[0024] Fourthly, embodiments of this application provide a model training apparatus, including: a multi-scale feature processing module, used to perform multi-scale feature processing on a second image through a feature extraction network to obtain a second multi-scale feature sequence;

[0025] The first feature reconstruction module is used to reconstruct the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence in the i-th layer feature reconstruction network by each of the M attention networks in the i-th layer feature reconstruction network, to obtain the fourth reconstructed feature sequence. Here, the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, and the input feature sequence of the 1-th layer feature reconstruction network is the second multi-scale feature sequence, i = 1, 2...N, where N and M are both positive integers.

[0026] The second feature reconstruction module is used to normalize the fourth reconstructed feature sequence through a summation and normalization layer if M equals 1, to obtain the fifth reconstructed feature sequence; or, if M is greater than 1, to perform a weighted summation of the M fourth reconstructed feature sequences through a summation and normalization layer, and to normalize the weighted summation result to obtain the fifth reconstructed feature sequence.

[0027] The third feature reconstruction module is used to process the fifth reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain the sixth reconstructed feature sequence;

[0028] The loss calculation module is used to calculate the loss of the target model consisting of the feature extraction network and the N-layer feature reconstruction network based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the Nth layer feature reconstruction network.

[0029] The training module is used to train the target model and the reference feature sequence for each layer in the N-layer feature reconstruction network based on the loss of the target model.

[0030] In some implementations, the multi-scale feature processing module is specifically used to: extract multi-scale features from the second image using a pre-trained model in the feature extraction network to obtain a fifth feature map at multiple scales; fuse the fifth feature map at multiple scales using a fusion layer in the feature extraction network to obtain a second fused feature map; transform the shape of the second fused feature map using a transformation layer in the feature extraction network to obtain a third multi-scale feature sequence, and obtain a second multi-scale feature sequence based on the third multi-scale feature sequence.

[0031] In some implementations, the multi-scale feature processing module is specifically used to: perform average pooling operations on the fifth feature maps at multiple scales through a fusion layer to obtain multiple sixth feature maps at the same scale; and merge the multiple sixth feature maps at the same scale along the channel dimension through a fusion layer to obtain a second fused feature map.

[0032] In some implementations, the multi-scale feature processing module is specifically used to: perturb the third multi-scale feature sequence to obtain the second multi-scale feature sequence.

[0033] In some implementations, the loss calculation module is specifically used to: restore the sixth reconstructed feature sequence output by the Nth layer feature reconstruction network to the seventh feature map at multiple scales; restore the second multi-scale feature sequence to the eighth feature map at multiple scales; and calculate the loss of the target model based on the seventh and eighth feature maps at multiple scales.

[0034] In some implementations, the loss calculation module is specifically used to: calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale; and sum the losses of the target model at each scale to obtain the total loss of the target model.

[0035] Fifthly, an electronic device is provided, comprising: a processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory, and performing methods as described in the first to second aspects or their respective implementations.

[0036] Sixthly, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods described in the first to second aspects or their respective implementations.

[0037] A seventh aspect provides a computer program product including computer program instructions that cause a computer to perform the methods described in the first to second aspects or their respective implementations.

[0038] Eighthly, a computer program is provided that causes a computer to perform the methods described in the first to second aspects or their respective implementations.

[0039] The technical solution provided in this application reduces the possibility of learning shortcuts by eliminating residual connections above the attention network of the feature extraction network. Furthermore, this application proposes introducing a reference feature sequence into each layer of the feature reconstruction network. Since this reference feature sequence conforms to normal image feature patterns, the similarity between abnormal features in the input feature sequence and the reference feature sequence is small, while the similarity between normal features in the input feature sequence and the reference feature sequence is large. This allows the first attention matrix corresponding to each feature in the input feature sequence to reflect its differences from different reference features, thereby reducing the possibility of rank collapse of the attention matrix and improving the reliability of anomaly detection. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram of a feature reconstruction network for a Transformer structure;

[0042] Figure 2 This is a schematic diagram of a system architecture according to an embodiment of this application;

[0043] Figure 3 A flowchart illustrating an anomaly detection method provided in this application embodiment;

[0044] Figure 4 A schematic diagram of a target model provided in an embodiment of this application;

[0045] Figure 5 This is a schematic diagram of average pooling;

[0046] Figure 6 This is a schematic diagram of a first fusion feature map;

[0047] Figure 7 A schematic diagram of a feature extraction network provided in an embodiment of this application;

[0048] Figure 8 A schematic diagram of the first attention network provided in an embodiment of this application;

[0049] Figure 9 A schematic diagram of feature reconstruction provided in an embodiment of this application;

[0050] Figure 10 A schematic diagram of the second attention network provided in an embodiment of this application;

[0051] Figure 11 Another feature reconstruction diagram provided for an embodiment of this application;

[0052] Figure 12 Schematic diagrams of other network layers provided in the embodiments of this application;

[0053] Figure 13 A schematic diagram illustrating the calculation of the feature difference score map provided in the embodiments of this application;

[0054] Figure 14 A schematic diagram of another target model provided in an embodiment of this application;

[0055] Figure 15 A flowchart illustrating a model training method provided in this application embodiment;

[0056] Figure 16 A schematic diagram of another feature extraction network provided in an embodiment of this application;

[0057] Figure 17 This is a schematic diagram illustrating the calculation of the target model loss provided in an embodiment of this application;

[0058] Figure 18 A schematic diagram of another target model provided in an embodiment of this application;

[0059] Figure 19 A schematic diagram of an anomaly detection device 1900 provided in an embodiment of this application;

[0060] Figure 20 A schematic diagram of a model training device 2000 provided in an embodiment of this application;

[0061] Figure 21 This is a schematic block diagram of the electronic device 2100 provided in the embodiments of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0064] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0065] As mentioned above, the current feature reconstruction network is a Transformer structure, which suffers from the learning shortcut problem. In other words, this structure does not easily learn normal image feature patterns. As a result, for actual abnormal images, the feature reconstruction network of this structure is not easy to reconstruct feature sequences that conform to normal image feature patterns. This leads to the problem that the reliability of anomaly detection based on the difference between multi-scale feature sequences and reconstructed feature sequences is low.

[0066] Figure 1 A schematic diagram of a feature reconstruction network for a Transformer structure, as shown below. Figure 1 As shown, this feature reconstruction network consists of N encoder layers and N decoder layers. Each encoder layer includes an attention network, a feed-forward network (FFN), and residual and normalization (ADD & Normalize) layers. Each decoder layer also includes an attention network, a feed-forward network, and residual and normalization layers.

[0067] In this encoder layer, the attention network calculates the query (Q) vector, key (K) vector, and value (V) vector for each feature in the input feature sequence using three preset weight matrices WQ, WK, and WV. Based on the Q and K vectors, it calculates the attention matrix for each feature. Positional information can be incorporated into this calculation, and the attention matrix is ​​multiplied by the V vector to obtain the product. The residual and normalization layer of this encoder layer sums the product and the input feature sequence of that layer, then normalizes the sum. The feedforward network of this encoder layer performs linear and nonlinear transformations on the normalized result, and after another residual and normalization layer, the encoded sequence corresponding to the input feature sequence of that layer is obtained. Finally, the encoded sequence output by the Nth encoder layer is used as the input feature sequence of the Nth decoder.

[0068] In this process, the attention network, residual, and normalization layers in each decoder layer process multi-scale feature sequences to obtain a normalized result. Positional information can be incorporated when calculating the attention matrix. Furthermore, the input feature sequence of this layer can be merged with the normalized result, and the merged result is processed again through the same attention network and residual / normalization layers in the decoder layer to obtain another normalized result. Even further, the feedforward network of this decoder layer can perform linear and nonlinear transformations on the normalized result, which is then passed through another residual / normalization layer to obtain the decoded sequence corresponding to the input feature sequence of this layer. Finally, the decoded sequence output by the Nth layer decoder can be used as the reconstructed feature sequence corresponding to the multi-layer feature sequences.

[0069] It should be understood that the learning shortcut problem currently introduced by the Transformer structure is mainly caused by the residual connections in this structure, such as... Figure 1 As shown, this is especially due to the residual connections above the attention network. This is because the residual connections cause the output of the attention network to include the input of the attention network, forming a learning shortcut. This makes it difficult for the attention network to learn the working mode from input to output. In anomaly detection scenarios, this prevents the attention network from learning normal image feature patterns. In other words, it prevents the Transformer structure from learning normal image feature patterns. As a result, for actual abnormal images, the feature reconstruction network of this structure is not easy to reconstruct feature sequences that conform to normal image feature patterns. This leads to the problem of low reliability of anomaly detection based on the difference between multi-scale feature sequences and reconstructed feature sequences.

[0070] To address the aforementioned technical issues, this application proposes omitting residual connections above the attention network in the feature extraction network. Considering that residual connections help prevent rank collapse of the attention matrix, this application proposes introducing a reference feature sequence at each layer. Since this reference feature sequence conforms to normal image feature patterns, the similarity between abnormal features in the input feature sequence and the reference feature sequence is small, while the similarity between normal features in the input feature sequence and the reference feature sequence is large. This allows the attention matrix corresponding to each feature in the input feature sequence to reflect its differences from different reference features, thereby reducing the possibility of rank collapse of the attention matrix and improving the reliability of anomaly detection.

[0071] In some embodiments of this application, the system architecture is as follows: Figure 2 As shown.

[0072] Figure 2 This is a schematic diagram of a system architecture according to an embodiment of this application. The system architecture includes user equipment 201, data acquisition equipment 202, training equipment 203, execution equipment 204, database 205, and content library 206.

[0073] The data acquisition device 202 is used to read training data from the content library 206 and store the read training data in the database 205. The training data involved in this embodiment includes normal images.

[0074] The training device 203 trains the target model, which consists of a feature extraction network and an N-layer feature reconstruction network, based on the training data maintained in the database 205, so that the trained target model can effectively reconstruct features.

[0075] Additionally, refer to Figure 2 The execution device 204 is equipped with an I / O interface 207 for data interaction with external devices. For example, it can receive images sent by the user device 201 through the I / O interface. The computation module 208 in the execution device 204 uses the feature extraction module in the trained target model to perform feature processing on the input image and outputs the multi-scale feature sequence corresponding to the image. Then, the N-layer feature reconstruction network in the target model can reconstruct the multi-scale feature sequence to obtain the reconstructed feature sequence. Finally, the anomaly detection module 209 in the computation module 208 determines the feature difference score map of the image based on the reconstructed feature sequence and the multi-scale feature sequence. Based on the feature difference score map of the image, it determines the abnormal pixels of the image. Finally, the anomaly detection results, such as the coordinates of the abnormal pixels, are sent to the user device 201 through the I / O interface.

[0076] User equipment 201 may include mobile phones, tablets, laptops, handheld computers, mobile internet devices (MIDs), desktop computers, or other terminal devices with browser installation capabilities.

[0077] The execution device 204 can be a server.

[0078] For example, the server can be a rack server, blade server, tower server, or cabinet server, etc. The server can be a standalone test server or a test server cluster composed of multiple test servers.

[0079] In this embodiment, the execution device 204 is connected to the user equipment 201 via a network. The network can be an intranet, the Internet, Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), 4G, 5G, Bluetooth, Wi-Fi, or a voice communication network, or other wireless or wired networks.

[0080] It should be noted that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the figure do not constitute any limitation. In some embodiments, the data acquisition device 202, user device 201, training device 203, and execution device 204 may be the same device. The database 205 may be distributed across one server or multiple servers, and the content library 206 may be distributed across one server or multiple servers.

[0081] The embodiments of this application will be described in detail below:

[0082] Figure 3 This is a flowchart illustrating an anomaly detection method provided in an embodiment of this application. This method can be executed by an execution device, which may be... Figure 2 The execution device 204 in the middle, but not limited to it, such as Figure 3 As shown, the method may include:

[0083] S310: Perform multi-scale feature processing on the first image through a feature extraction network to obtain the first multi-scale feature sequence;

[0084] S320: Through each of the M attention networks in the i-th layer feature reconstruction network, the input feature sequence of the i-th layer feature reconstruction network is reconstructed based on the reference feature sequence in the i-th layer feature reconstruction network to obtain the first reconstructed feature sequence. Here, the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, and the input feature sequence of the 1-th layer feature reconstruction network is the first multi-scale feature sequence, i = 1, 2... N, where N and M are both positive integers, and the reference feature sequence in the i-th layer feature reconstruction network conforms to the normal image feature pattern.

[0085] S330: If M equals 1, the first reconstructed feature sequence is normalized through a summation and normalization layer to obtain the second reconstructed feature sequence; or, if M is greater than 1, the M first reconstructed feature sequences are weighted and summed through a summation and normalization layer, and the weighted summation result is normalized to obtain the second reconstructed feature sequence.

[0086] S340: The second reconstructed feature sequence is processed by other network layers in the i-th layer feature reconstruction network to obtain the third reconstructed feature sequence;

[0087] S350: Based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the Nth layer feature reconstruction network, determine the feature difference score map of the first image;

[0088] S360: Based on the feature difference score map of the first image, determine the abnormal pixels of the first image.

[0089] In this embodiment, the model consisting of the feature extraction network and the N-layer feature reconstruction network can be referred to as the target model. The structure of the target model will be described below:

[0090] Figure 4 A schematic diagram of a target model provided in an embodiment of this application, such as... Figure 4 As shown, the feature extraction network can perform multi-scale feature processing on the image to obtain the first multi-scale feature sequence; each feature extraction network in the N-layer feature reconstruction network includes: an attention network, a summation and normalization layer, and other network layers, where, Figure 4Taking a feature reconstruction network with two attention networks per layer as an example, each attention network can reconstruct the input feature sequence of the feature reconstruction network based on the reference feature sequence in the feature reconstruction network of that layer, to obtain the first reconstructed feature sequence. If M equals 1, the summation and normalization layer in each feature reconstruction network normalizes the first reconstructed feature sequence to obtain the second reconstructed feature sequence. If M is greater than 1, the summation and normalization layer in each feature reconstruction network performs a weighted summation on the M first reconstructed feature sequences and normalizes the weighted summation result to obtain the second reconstructed feature sequence. The other network layers in each feature reconstruction network process the second reconstructed feature sequence to obtain the third reconstructed feature sequence.

[0091] The following is a detailed explanation of S310:

[0092] In some implementations, the execution device can capture an image of the object to be detected using an image acquisition device to obtain a first image. This image acquisition device can be an internal image acquisition unit of the execution device, such as a camera, or it can be a peripheral device of the execution device; this application embodiment does not impose any limitations on this.

[0093] It should be understood that S310 can be implemented in any of the following ways, but is not limited to:

[0094] In the first implementation, S310 may include:

[0095] S310-1A: Multi-scale feature extraction is performed on the first image using a pre-trained model in the feature extraction network to obtain the first feature map under multiple scales;

[0096] In some implementations, the pre-trained model can be a Convolutional Neural Network (CNN) containing multiple convolutional layers, such as ResNet-50, ResNet-34, EfficientNet, VGG-16O, etc. Each convolutional layer in this CNN can extract key feature information from the first image, which is output as a feature map, where each element is called a feature point. In this embodiment, the feature map extracted from the first image through each convolutional layer is referred to as the first feature map.

[0097] It should be understood that each first feature map is the result of processing the first image through a specific convolutional kernel, i.e., a filter, which can capture certain features or patterns of the first image. Different convolutional kernels can capture different types of features, such as edges, corners, and textures. As the number of network layers increases, the "depth" of the feature maps also increases. Shallower feature maps tend to capture more basic and general features, such as edges and simple textures, while deeper feature maps can represent more complex and high-level features, such as specific parts of an object or complex shapes.

[0098] It should be understood that if the first image is a two-dimensional image, then the feature map obtained after processing by the convolutional kernel will still be a two-dimensional structure. However, in convolutional networks, to capture more features, each convolutional layer typically uses multiple convolutional kernels. Each convolutional kernel generates a two-dimensional feature map, and these feature maps are stacked together to form a three-dimensional feature volume, where the third dimension (depth) corresponds to different numbers of convolutional kernels. Therefore, the first feature map output by each convolutional layer can be a three-dimensional structure. Let f be the first feature map output by the i-th convolutional layer. i , then f i ∈R Hi *Wi*Di H i and W i These represent the first feature map f respectively. i Height and width, D i This represents the number of convolutional kernels or channels in the i-th convolutional layer.

[0099] S310-2A: The first feature map is obtained by fusing the first feature maps at multiple scales through the fusion layer in the feature extraction network;

[0100] It should be understood that fusing the first feature maps at multiple scales can enhance image features, thereby improving the robustness of anomaly detection.

[0101] In some implementations, the execution device can perform average pooling operations on the first feature maps at multiple scales through a fusion layer to obtain multiple second feature maps at the same scale; and then merge the multiple second feature maps at the same scale along the channel dimension through the fusion layer to obtain a first fused feature map.

[0102] It should be understood that in average pooling, the average of all values ​​within the pooling window is used as the output of that window. Average pooling helps preserve the background information of the image because it takes into account all pixel values ​​within the window.

[0103] Average pooling includes the following steps:

[0104] 1. Define the pooled window:

[0105] Select a window of a fixed size (such as 2*2, 3*3, etc.), and the window will slide on the feature map.

[0106] 2. Define the step size:

[0107] Define the stride of the pooling window as it slides across the feature map. The stride is usually the same as the window size, but it can also be different.

[0108] 3. Padding:

[0109] In some cases, to maintain the size of the feature map, additional zero values ​​can be added to the edges of the feature map (called zero padding).

[0110] 4. Execution pooling:

[0111] For each location on the feature map, a pooling window is placed at that location, and the value within the window is calculated based on the average pooling type. The result of this calculation is then placed at the corresponding location in the output feature map.

[0112] For example, Figure 5 This is a schematic diagram of average pooling, such as... Figure 5 As shown, for Figure 5 The left-hand feature map is processed by selecting a 2x2 pooling window, sliding the window with a stride of 2x2, and calculating the average of all values ​​within each window. The final output is the right-hand feature map. Figure 5 The feature map on the left is the first feature map, and the feature map on the right is the second feature map.

[0113] Suppose the first feature map f i The second feature map obtained after average pooling is denoted as g. i And let the first fused feature map be denoted as G, then g i ∈R H*W*Di , G∈R H*W*D Where H and W represent the second feature map g, respectively. i Height and width, D i This represents the number of convolutional kernels or channels in the i-th convolutional layer, and D represents the sum of the number of convolutional kernels or channels in all second feature maps. For example, Figure 6 This is a schematic diagram of a first fused feature map, such as... Figure 6 As shown, the length, width, and height of the first fused image are H, W, and D, respectively.

[0114] In some implementations, the execution device can perform padding or cropping operations on the first feature maps at multiple scales through a fusion layer to obtain multiple second feature maps at the same scale; and then merge the multiple second feature maps at the same scale along the channel dimension through the fusion layer to obtain a first fused feature map.

[0115] For example, assuming the size of the second feature map is specified as 4*4, for a first feature map with a size smaller than 4*4, feature points with a value of 0 can be filled at its edges to make its size 4*4. For a first feature map with a size greater than 4*4, its edge feature points can be clipped to make its size 4*4. For a first feature map with a size equal to 4*4, no operation can be performed on it.

[0116] It should be understood that the embodiments of this application do not limit the calculation method of the first fused feature map.

[0117] S310-3A: The first fused feature map is transformed by the transformation layer in the feature extraction network to obtain the first multi-scale feature sequence.

[0118] In some possible implementations, assuming the first fused image is a three-dimensional structure of H*W*D, the execution device can convert the first fused feature image into H*W D-dimensional vectors and sort the H*W D-dimensional vectors in a preset order to obtain the first multi-scale feature sequence.

[0119] In some implementations, the preset order is first from left to right and then from top to bottom, but it is not limited to this.

[0120] Figure 7 This is a schematic diagram of a feature extraction network provided in an embodiment of this application, as shown below. Figure 7 As shown, the first feature map under multiple scales is processed to form a first multi-scale feature sequence, wherein each box in the first multi-scale feature sequence represents a D-dimensional vector.

[0121] In the second implementation method, S310 may include:

[0122] S310-1B: Input the first image and the size requirements that the first multi-scale feature sequence needs to meet into the neural network model to obtain the first multi-scale feature sequence.

[0123] In some implementations, the neural network model can be trained using training samples. Each training sample may include: an image, the size requirements that the multi-scale feature sequence corresponding to the image must meet, and the actual multi-scale feature sequence, where the actual multi-scale feature sequence can serve as a sample label. The training device can employ supervised training; for example, it can input the image and the size requirements that the multi-scale feature sequence corresponding to the image must meet into the neural network model and output a predicted multi-scale feature sequence. Furthermore, the training device can calculate a loss based on the actual and predicted multi-scale feature sequences included in all training samples, and adjust the parameters of the neural network model based on this loss until the training iterations reach a preset number or the loss reaches its minimum value, at which point training stops.

[0124] In some implementations, the training device may use any of the following loss functions when training a neural network model, but is not limited to: L1 loss function, mean squared error (MSE) loss function, cross-entropy loss function, etc.

[0125] Based on this, the execution device can input the first image and the size requirements that the first multi-scale feature sequence needs to meet into the neural network model to obtain the first multi-scale feature sequence.

[0126] The S320 will be described in detail below:

[0127] In some implementations, the M attention networks include at least one of the following: a first attention network and a second attention network; correspondingly, the input feature sequence of the i-th layer feature reconstruction network is reconstructed based on the reference feature sequence in the i-th layer feature reconstruction network through each of the M attention networks in the i-th layer feature reconstruction network, to obtain the first reconstructed feature sequence, which includes at least one of the following:

[0128] 1. Using the first attention network, each feature in the input feature sequence of the i-th layer feature reconstruction network is converted into a first query vector and a first value vector, and each feature in the reference feature sequence of the i-th layer feature reconstruction network is converted into a first key vector; based on the first query vector and the first key vector, the first attention matrix corresponding to each feature in the input feature sequence of the i-th layer feature reconstruction network is calculated; based on the first attention matrix and the first value vector, the first reconstructed feature sequence is calculated.

[0129] 2. Using the second attention network, each feature in the input feature sequence of the i-th layer feature reconstruction network is converted into a second query vector, and each feature in the reference feature sequence of the i-th layer feature reconstruction network is converted into a second key vector and a second value vector; based on the second query vector and the second key vector, the second attention matrix corresponding to each feature in the input feature sequence of the i-th layer feature reconstruction network is calculated; based on the second attention matrix and the second value vector, the first reconstructed feature sequence is calculated.

[0130] The following section describes the case where the first reconstructed feature sequence is obtained through the first attention network:

[0131] For example, Figure 8 A schematic diagram of the first attention network provided in the embodiments of this application, as shown below. Figure 8 As shown, the execution device can use a first attention network to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a first query vector and a first value vector, and convert each feature in the reference feature sequence of the i-th layer feature reconstruction network into a first key vector. For each feature in the input feature sequence, a first attention matrix can be calculated based on the first query vector corresponding to the feature and the first key vector corresponding to each feature in the input feature sequence. In other words, each feature in the input feature sequence corresponds to a first attention matrix. Further, the product of the first attention matrix and the matrix formed by all the first value vectors can be calculated to obtain the reconstructed feature corresponding to the feature. The reconstructed features corresponding to all the features in the input feature sequence then form the first reconstructed feature sequence.

[0132] It should be understood that, as Figure 8 As shown, the input feature sequence in the first layer feature reconstruction network is the first multi-scale feature sequence, where each box represents a feature, and the dimension of the reference feature sequence of each layer is the same as the dimension of the input feature sequence of that layer.

[0133] It should be understood that, for each feature in the input feature sequence, the execution device performs the following operation on the product of the first attention matrix corresponding to the feature and the matrix composed of all first value vectors through the first attention network: the execution device multiplies each attention value in the first attention matrix with the first value vector at the corresponding position in the matrix composed of all first value vectors through the first attention network to obtain the multiplication result at each position, and then adds these multiplication results to obtain the reconstructed feature corresponding to the feature.

[0134] It should be understood that since the first key vector is derived from the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, when calculating the first attention matrix based on the first key vector and the first query vector, the abnormal features in the input feature sequence, i.e. the features corresponding to the abnormal regions, have a low similarity to the first key vector. Therefore, the first reconstructed feature sequence usually does not retain the abnormal image feature pattern, but most likely retains the normal image feature pattern, which makes the difference between the input feature sequence and the reconstructed feature sequence larger, thus facilitating anomaly detection.

[0135] In some implementations, for each feature in the input feature sequence, the execution device can compute a query weight matrix WQ and multiply it by the feature, i.e., a D-dimensional vector, through a first attention network to obtain the first query vector corresponding to the feature.

[0136] In some implementations, for each feature in the input feature sequence, the execution device can compute a value weight matrix WV and multiply it by the feature, i.e., a D-dimensional vector, through a first attention network to obtain the first value vector corresponding to the feature.

[0137] In some implementations, for each feature in the reference feature sequence, the execution device can compute a key weight matrix WK and multiply it by the feature, i.e., a D-dimensional vector, through a first attention network to obtain the first key vector corresponding to the feature.

[0138] Suppose that each feature in the input feature sequence of the i-th layer feature reconstruction network is called the first feature. Then, the execution device can obtain the first attention matrix corresponding to the first feature through the first attention network in any of the following possible ways, but is not limited to:

[0139] In some implementations, the execution device uses a first attention network to perform a scaled dot product between the first query vector corresponding to the first feature and the transpose of each first key vector, and then performs normalization processing to obtain the first attention matrix corresponding to the first feature.

[0140] Assuming we denote the first query vector corresponding to the first feature as Q1, the first key vector corresponding to each feature as K1, and the matrix formed by the transpose of these first key vectors K1 as K, then the first attention matrix is... Where, d K1 Let K1 denote the dimension of K1, where Q1·K is a matrix in which each element represents the dot product of Q1 and the transpose of K1.

[0141] In some implementations, the execution device uses a first attention network to perform a scaled dot product between the first query vector corresponding to the first feature and the transpose of each first key vector to obtain a third attention matrix corresponding to the first feature; and calculates the sum of the third attention matrix corresponding to the first feature and the first mask matrix corresponding to the first feature, and performs normalization processing to obtain a first attention matrix corresponding to the first feature; wherein, the value of the position corresponding to the first feature and the surrounding positions in the first mask matrix are negative infinity, and the value of the remaining positions is 0.

[0142] The first mask matrix, also known as the neighbor masking attention matrix or attention graph, is used to mask the attention values ​​of the first feature and its neighboring features in the third attention matrix corresponding to the first feature.

[0143] In some possible implementations, the neighbor features of the first feature may include at least one of the following, but not limited to: the neighbor features directly above the first feature, the neighbor features directly below the first feature, the neighbor features to the left of the first feature, the neighbor features to the right of the first feature, the neighbor features to the top left of the first feature, the neighbor features to the bottom left of the first feature, the neighbor features to the top right of the first feature, and the neighbor features to the bottom right of the first feature.

[0144] It should be understood that the surrounding positions of the first feature refer to the corresponding positions of the neighboring features of the first feature in the first mask matrix.

[0145] Assuming we denote the first query vector as Q1, the first key vector as K1, the matrix formed by the transposes of these first key vectors K1 as K, and the first attention matrix as M1, then the first attention matrix is... Where, d K1 Let K1 be the dimension.

[0146] For example, Figure 9 A schematic diagram of feature reconstruction provided for an embodiment of this application, such as... Figure 9 As shown, for the first attention network in the i-th layer feature reconstruction network, its input feature sequence and reference feature sequence can be represented in matrix form, where each box in the matrix can represent a feature, which can be a D-dimensional vector. The execution device can use this first attention network to convert the first feature into Q1, convert each feature in the input feature sequence into V1, and convert each feature in the reference feature sequence into K1. Further, the execution device can use the first attention network to perform a scaled dot product of Q1 and the transpose of each K1 to obtain the third attention matrix A1; and calculate the sum of the third attention matrix A1 and the first mask matrix M1, and normalize it using the Softmax function to obtain the first attention matrix A2; where the value of the first feature in M1 and its surrounding positions is negative infinity, and the value of other positions is 0, as shown. Figure 9As shown, the values ​​of the top, bottom, left, right, top-left, bottom-left, top-right, and bottom-right neighbor features of the first feature are all negative infinity, while the values ​​of the other features are 0. Furthermore, the execution device can use the first attention network to calculate the product of the first attention matrix A2 and the matrix formed by all V1 to obtain the reconstructed feature corresponding to the first feature.

[0147] It should be understood that the problem of learning shortcuts is mainly caused by the fact that the output of the network directly includes the input of the network. Based on this, the embodiments of this application propose to mask the attention values ​​corresponding to the first feature and its neighboring features in the third attention matrix corresponding to the first feature, so that when reconstructing the first feature, the first value vector corresponding to the first feature and its neighboring features can be ignored, thereby preventing the reconstructed feature from including the first feature and its neighboring features, reducing the possibility of learning shortcuts, and thus improving the robustness of the anomaly detection method.

[0148] The following section explains the case where the first reconstructed feature sequence is obtained through the second attention network:

[0149] For example, Figure 10 A schematic diagram of the second attention network provided in the embodiments of this application, as shown below. Figure 10 As shown, the execution device can use a second attention network to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a second query vector, and convert each feature in the reference feature sequence of the i-th layer feature reconstruction network into a second key vector and a second value vector. For each feature in the input feature sequence, a second attention matrix can be calculated based on the second query vector corresponding to the feature and the second key vector corresponding to each feature in the reference feature sequence. In other words, each feature in the input feature sequence corresponds to a second attention matrix. Further, the product of the second attention matrix and the matrix formed by all the second value vectors can be calculated to obtain the reconstructed feature corresponding to the feature. The reconstructed features corresponding to all the features in the input feature sequence then form the first reconstructed feature sequence.

[0150] It should be understood that, as Figure 10 As shown, the input feature sequence in the first layer feature reconstruction network is the first multi-scale feature sequence, where each box represents a feature, and the dimension of the reference feature sequence of each layer is the same as the dimension of the input feature sequence of that layer.

[0151] It should be understood that, for each feature in the input feature sequence, the execution device performs the following operation through the second attention network: multiplying each attention value in the second attention matrix with the corresponding second value vector in the matrix formed by all second value vectors, obtaining the multiplication result at each position, and then adding these multiplication results to obtain the reconstructed feature corresponding to that feature.

[0152] It should be understood that since the second key vector is derived from features in the reference feature sequence, and the reference feature sequence conforms to normal image feature patterns, when calculating the second attention matrix based on the second key vector and the second query vector, the anomalous features in the input feature sequence—that is, the features corresponding to the anomalous regions—have low similarity to the second key vector. Therefore, the first reconstructed feature sequence usually does not retain anomalous image feature patterns, but rather retains normal image feature patterns, resulting in a larger difference between the input feature sequence and the reconstructed feature sequence, which is beneficial for anomaly detection. Furthermore, since the second value vector is also derived from features in the reference feature sequence, and the reference feature sequence conforms to normal image feature patterns, the first reconstructed feature sequence generated when reconstructing features based on the second value vector is more likely to retain normal image feature patterns, resulting in a larger difference between the input feature sequence and the reconstructed feature sequence, further improving the robustness of the anomaly detection method.

[0153] In some implementations, for each feature in the input feature sequence, the execution device can compute a query weight matrix WQ and multiply it by the feature, i.e., a D-dimensional vector, through a second attention network to obtain the second query vector corresponding to the feature.

[0154] In some implementations, for each feature in the reference feature sequence, the execution device can compute a value weight matrix WV and multiply it by the feature, i.e., a D-dimensional vector, through a second attention network to obtain the second value vector corresponding to the feature.

[0155] In some implementations, for each feature in the reference feature sequence, the execution device can compute a key weight matrix WK and multiply it by the feature, i.e., a D-dimensional vector, through a second attention network to obtain the second key vector corresponding to the feature.

[0156] Suppose that each feature in the input feature sequence of the i-th layer feature reconstruction network is called a second feature. Then, the execution device can obtain the second attention matrix corresponding to the second feature through the second attention network in any of the following possible ways, but is not limited to:

[0157] In some implementations, the execution device uses a second attention network to perform a scaled dot product between the second query vector corresponding to the second feature and the transpose of each second key vector, and then performs normalization processing to obtain the second attention matrix corresponding to the second feature.

[0158] Suppose we denote the second query vector corresponding to the second feature as Q2, the second key vector corresponding to each feature as K2, and the matrix formed by the transpose of these second key vectors K2 as K', then the second attention matrix is... Where, d K2 Let Q2 represent the dimension of K2, where Q2·K' is a matrix in which each element represents the dot product of Q2 and the transpose of K2.

[0159] In some implementations, the execution device uses a second attention network to perform a scaled dot product between the second query vector corresponding to the second feature and the transpose of each second key vector to obtain a fourth attention matrix corresponding to the second feature; and calculates the sum of the fourth attention matrix corresponding to the second feature and the second mask matrix corresponding to the second feature, and performs normalization processing to obtain a second attention matrix corresponding to the second feature; wherein, the value of the position corresponding to the second feature and the surrounding positions in the second mask matrix is ​​0, and the value of the remaining positions is negative infinity.

[0160] The second mask matrix is ​​used to mask the attention values ​​of other features in the fourth attention matrix corresponding to the second feature, excluding the second feature and its neighboring features.

[0161] In some possible implementations, the neighbor features of the second feature may include at least one of the following, but not limited to: the neighbor features directly above the second feature, the neighbor features directly below the second feature, the neighbor features to the left of the second feature, the neighbor features to the right of the second feature, the neighbor features to the top left of the second feature, the neighbor features to the bottom left of the second feature, the neighbor features to the top right of the second feature, and the neighbor features to the bottom right of the second feature.

[0162] It should be understood that the surrounding positions of the second feature refer to the corresponding positions of the neighboring features of the second feature in the second mask matrix.

[0163] Assuming we denote the second query vector as Q2, the second key vector as K2, the matrix formed by the transposes of these second key vectors K2 as K', and the second attention matrix as M2, then the second attention matrix is... Where, d K2 This represents the dimension of K2.

[0164] For example, Figure 11 Another feature reconstruction diagram provided for an embodiment of this application, such as Figure 11As shown, for the second attention network in the i-th layer feature reconstruction network, its input feature sequence and reference feature sequence can be represented in matrix form. Each box in the matrix represents a feature, which can be a D-dimensional vector. The execution device can use this second attention network to convert the second feature into Q2, and each feature in the reference feature sequence into V2, and each feature in the reference feature sequence into K2. Further, the execution device can use the second attention network to perform a scaled dot product of Q2 and the transpose of each K2 to obtain the fourth attention matrix A3. The sum of the fourth attention matrix A3 and the second mask matrix M2 is calculated and normalized using the Softmax function to obtain the second attention matrix A4. In M2, the values ​​at the corresponding positions of the second feature and their surrounding positions are 0, and the values ​​at other positions are negative infinity. Figure 11 As shown, the values ​​of the corresponding positions of the top, bottom, left, right, top left, bottom left, top right, and bottom right neighbor features of the second feature are all 0, while the values ​​of the remaining positions are negative infinity. Furthermore, the execution device can use the second attention network to calculate the product of the second attention matrix A2 and the matrix formed by all V2 to obtain the reconstructed feature corresponding to the second feature.

[0165] It should be understood that since the second value vector is the result of feature transformation in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, this application embodiment proposes to mask the attention values ​​corresponding to other features in the fourth attention matrix corresponding to the second feature, except for the second feature and its neighboring features, so that when reconstructing the second feature, the second value vector corresponding to the second feature and its neighboring features can be referenced in particular, thereby further improving the robustness of the anomaly detection method.

[0166] It should be understood that although the first attention network solves the problem of learning shortcuts to some extent, it may still leave abnormal features in its output first reconstructed feature sequence. This is because the value vector in the first attention network corresponds to the features in the input feature sequence, and there may be abnormal features in the input feature sequence. In order to solve this problem, this application proposes to reconstruct features through a second attention network. This is because the value vector in the second attention network corresponds to the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern. As a result, the possibility of abnormal features in the first reconstructed feature sequence output by the second attention network is lower, thereby improving the robustness of the anomaly detection method.

[0167] It should be understood that the attention mechanism used in this application embodiment for the first attention network combined with the first mask matrix can be called Masked Learnable Key Attention (MLKA), and the attention mechanism used in the second attention network combined with the second mask matrix can be called Local Cross Attention (LCA). This application embodiment may employ at least one of these two attention mechanisms.

[0168] It should be understood that by combining MLKA and LCA, on the one hand, since the value vector in the LCA-based attention network corresponds to the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, the probability of abnormal features in the first reconstructed feature sequence output by the attention network is low, thereby improving the robustness of the anomaly detection method. On the other hand, since the attention values ​​corresponding to other features in the fourth attention matrix corresponding to the second feature, except for the second feature and its neighboring features, are masked in the LCA-based attention network, when reconstructing the second feature, the second value vector corresponding to the second feature and its neighboring features can be referenced more, thereby further improving the robustness of the anomaly detection method.

[0169] It should be understood that, when the first attention network and the second attention network are combined, or when the first attention network based on the MLKA mechanism and the first attention network based on the LCA mechanism are combined, for each feature in the input feature sequence, the first attention network can reconstruct the feature to obtain a reconstructed feature. Similarly, the second attention network can also reconstruct the feature to obtain a reconstructed feature. The summation and normalization layer can perform a weighted summation of the two reconstructed features and then perform normalization to obtain a reconstructed feature corresponding to that feature.

[0170] It should be noted that the first feature mentioned above can be every feature of the input feature sequence, and the second feature mentioned above can also be every feature of the input feature sequence. The first feature and the second feature can be the same feature or different features. For example, Figure 9 and Figure 10 The first feature and the second feature shown are not the same feature, but when the first attention network is combined with the second attention network, or when the first attention network based on the MLKA mechanism is combined with the first attention network based on the LCA mechanism, the first feature and the second feature refer to the same feature.

[0171] The S330 will be described in detail below:

[0172] In some implementations, if M equals 1, the execution device normalizes the first reconstructed feature sequence through a summation and normalization layer to obtain the second reconstructed feature sequence.

[0173] For example, if the N-layer feature reconstruction network only includes a first attention network using the MLKA mechanism, then the execution device will normalize the first reconstructed feature sequence output by the first attention network through the summation and normalization layer to obtain the second reconstructed feature sequence.

[0174] In some implementations, if M is greater than 1, the execution device performs a weighted summation on the M first reconstructed feature sequences through a summation and normalization layer to obtain a weighted summation result; and performs normalization processing on the weighted summation result to obtain a second reconstructed feature sequence.

[0175] For example, if an N-layer feature reconstruction network includes a first attention network using the MLKA mechanism and a second attention network using the LCA mechanism, then the execution device performs a weighted summation of the first reconstructed feature sequences output by the two attention networks through a summation and normalization layer to obtain a weighted summation result; and then performs normalization processing on the weighted summation result to obtain the second reconstructed feature sequence.

[0176] In some implementations, when the first reconstructed feature sequence output by the first attention network and the first reconstructed feature sequence output by the second attention network are weighted and summed through the summation and normalization layer, the weight corresponding to the first reconstructed feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstructed feature sequence output by the second attention network is a hyperparameter greater than 1.

[0177] For example, if an N-layer feature reconstruction network includes a first attention network using the MLKA mechanism and a second attention network using the LCA mechanism, then when performing a weighted summation of the first reconstructed feature sequences output by these two attention networks, the weight of the first attention network using the MLKA mechanism can be set to 1, and the weight of the second attention network using the LCA mechanism can be set to a hyperparameter α greater than 1, to ensure that the target model pays more attention to LCA.

[0178] It should be understood that the embodiments of this application do not limit the size of the weights corresponding to the first attention network using the MLKA mechanism and the weights corresponding to the second attention network using the LCA mechanism.

[0179] The S340 will be described in detail below:

[0180] In some implementations, the other network layers in the i-th layer feature reconstruction network include a feedforward network layer and another summing and normalization layer. Based on this, the execution device processes the second reconstructed feature sequence through the other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence. This process includes: the execution device performing linear and nonlinear transformations on the second reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain a first transformation result; then summing the first transformation result and the second reconstructed feature sequence through the summing and normalization layer; and finally normalizing the summing result to obtain the third reconstructed feature sequence.

[0181] For example, Figure 12 Schematic diagrams of other network layers provided in the embodiments of this application, such as Figure 12 As shown, the execution device obtains the third reconstructed feature sequence after processing through a feedforward network and a summation and normalization layer.

[0182] In some implementations, the other network layers in the i-th layer feature reconstruction network include a feedforward network layer. Based on this, the execution device processes the second reconstructed feature sequence through the other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence, including: the execution device performs linear and nonlinear transformations on the second reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain the third reconstructed feature sequence.

[0183] It should be understood that the embodiments of this application do not limit other network layers in the i-th layer feature reconstruction network, nor do they limit the corresponding processing of other network layers.

[0184] The S350 will be described in detail below:

[0185] It should be understood that S350 can be implemented in any of the following possible ways, but is not limited thereto:

[0186] In some implementations, S350 may include:

[0187] S350-1A: Reconstructs the third reconstructed feature sequence output by the Nth layer feature reconstruction network into a third feature map at multiple scales;

[0188] It should be understood that the third reconstructed feature sequence output by the Nth layer feature reconstruction network is the final reconstructed feature sequence corresponding to the first multi-scale feature sequence. Therefore, the feature sorting method of this third reconstructed feature sequence is the same as that of the first multi-scale feature sequence. The first multi-scale feature sequence is a sequence composed of H*W D-dimensional vectors. Based on this, the third reconstructed feature sequence is also a sequence composed of H*W D-dimensional vectors. Based on this, the execution device can convert this third reconstructed feature sequence into a feature map with an H*W*D three-dimensional structure. Then, according to the number and size of the convolutional kernels used in each convolutional layer of the N-layer feature reconstruction network, this feature map is converted into the feature map corresponding to each convolutional layer. The feature map corresponding to the i-th convolutional layer is an H*W*D feature map. i *W i *D i The three-dimensional structure.

[0189] S350-2A: Restore the first multi-scale feature sequence to the fourth feature map under multi-scale;

[0190] It should be understood that the first multi-scale feature sequence can be restored using the same method as the third reconstructed feature sequence described above to obtain the fourth feature map under multiple scales.

[0191] S350-3A: Calculate the feature difference scores of the third and fourth feature maps under multiple scales to obtain the feature difference score map of the first image.

[0192] It should be understood that the feature difference score map is also called the feature difference score matrix, and each element inside it represents the feature difference score of the corresponding pixel.

[0193] In some implementations, the execution device can use the mean squared error (MSE) method to calculate the feature difference scores of the third feature map and the fourth feature map at multiple scales to obtain the feature difference score map of the first image.

[0194] For example, suppose the j-th third feature map is denoted as Let the j-th fourth feature map be denoted as Let S be the feature difference score of the first image. Then, the feature difference score can be calculated using the following formula:

[0195]

[0196] Where L represents the number of convolutional layers in the feature extraction network, or the number of feature extraction layers. Indicates to and Subtracting corresponding elements and then squaring them, Upsample() is the upsampling function, which means... The corresponding matrix is ​​upsampled to the size of the first image.

[0197] In some possible implementations, the execution device can use mean squared error (MSE) and cosine similarity (Cos) to calculate the feature difference scores of the third feature map and the fourth feature map at multiple scales to obtain the feature difference score map of the first image, but is not limited thereto.

[0198] For example, suppose the feature difference between the j-th third feature map and the j-th fourth feature map in the feature difference score map of the first image is denoted as... Let the j-th third feature map be denoted as Let the j-th fourth feature map be denoted as Let S be the feature difference score of the first image. Then, the feature difference score can be calculated using the following formula:

[0199]

[0200] in, express and The corresponding element-wise dot products are summed, and L represents the feature extraction network.

[0201] The number of convolutional layers in the network, or more specifically, the number of feature extraction layers. Indicates to and Subtracting corresponding elements and then squaring them, Upsample() is the upsampling function, which means... The corresponding matrix is ​​upsampled to the size of the first image.

[0202] Figure 13 This is a schematic diagram illustrating the calculation of the feature difference score map provided in the embodiments of this application, as shown below. Figure 13 As shown, the execution device can use mean squared error (MSE) and cosine similarity (Cos) to calculate the feature difference scores of the third feature map and the fourth feature map under multiple scales, and obtain the feature difference score map of the first image.

[0203] It should be understood that the embodiments of this application do not limit the calculation method of the feature difference score map.

[0204] In some implementations, S350 may include:

[0205] S350-1B: Calculate the absolute value of the difference between the first multi-scale feature sequence and the third reconstructed feature sequence output by the Nth layer feature reconstruction network to obtain the feature difference score map.

[0206] In this feature difference score map, each score corresponds to a feature, which in turn corresponds to a region or location in the first image, and the region or location includes one or more pixels.

[0207] The S360 will be explained below:

[0208] In some implementations, the execution device can identify pixels in the feature difference score map that have a score higher than a preset score as abnormal pixels in the first image.

[0209] In some possible implementations, the preset score can be 0.7, 0.8, 0.9, etc., and this application embodiment does not limit this.

[0210] In some implementations, the execution device can identify pixels in the first image whose product of a score in the feature difference score map and a preset factor is higher than a preset score.

[0211] In some implementations, the size of the preset factor can be any real number greater than 0 and less than 1, and the embodiments of this application do not impose any restrictions on this.

[0212] It should be understood that the embodiments of this application do not limit the detection method for abnormal pixels.

[0213] The following will combine Figure 14 The anomaly detection method provided in the embodiments of this application will be described by way of example:

[0214] Let the model consisting of the feature extraction network and the N-layer feature reconstruction network be called the target model. Figure 14 A schematic diagram of another target model provided in the embodiments of this application, as shown below. Figure 14As shown, the execution device can perform multi-scale feature extraction on the image using a pre-trained model in the feature extraction network to obtain multi-scale feature maps. Further, the multi-scale feature maps are processed to obtain a first multi-scale feature sequence. Each feature extraction network in the N-layer feature reconstruction network includes: a first attention network and a second attention network, a summation and normalization layer, a feedforward network, and another summation and normalization layer. Specifically, the execution device can use the first attention network to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a first query vector and a first value vector, and convert each feature in the reference feature sequence of the i-th layer feature reconstruction network into a first key vector. For each feature in the input feature sequence, the execution device uses a first attention network to perform a scaled dot product between the first query vector corresponding to the feature and the transpose of each first key vector to obtain the third attention matrix corresponding to the feature. It then calculates the sum of the third attention matrix and the first mask matrix corresponding to the feature, and performs normalization to obtain the first attention matrix corresponding to the feature. Further, the execution device uses the first attention network to calculate the product of the first attention matrix corresponding to the feature and the matrix formed by all the first value vectors to obtain the reconstructed feature corresponding to the feature. The reconstructed features corresponding to each feature in the input feature sequence constitute the first reconstructed feature sequence. Similarly, the execution device uses a second attention network to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a second query vector, and converts each feature in the reference feature sequence of the i-th layer feature reconstruction network into a second key vector and a second value vector. For each feature in the input feature sequence, the execution device uses a second attention network to perform a scaled dot product between the second query vector corresponding to that feature and the transpose of each second key vector, obtaining a fourth attention matrix corresponding to that feature. The sum of the fourth attention matrix and the second mask matrix corresponding to that feature is calculated and normalized to obtain a second attention matrix corresponding to that feature. Further, the execution device uses the second attention network to calculate the product of the second attention matrix corresponding to that feature and the matrix formed by all second value vectors, obtaining a reconstructed feature corresponding to that feature. The reconstructed features corresponding to each feature in the input feature sequence constitute the first reconstructed feature sequence. Further still, a summation and normalization layer performs a weighted summation of the two first reconstructed feature sequences and normalizes the result to obtain a second reconstructed feature sequence. Finally, the execution device uses a feedforward network layer to perform linear and nonlinear transformations on the second reconstructed feature sequence to obtain the transformation result. Then, the summation and normalization layer sums the transformation result and the second reconstructed feature sequence, and normalizes the summation result to obtain a third reconstructed feature sequence.

[0215] This application provides an anomaly detection method, comprising: performing multi-scale feature processing on a first image through a feature extraction network to obtain a first multi-scale feature sequence; reconstructing the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence in the i-th layer feature reconstruction network through each of the M attention networks in the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, the input feature sequence of the first layer feature reconstruction network is the first multi-scale feature sequence, i = 1, 2...N, where N and M are both positive integers, and the reference feature sequence in the i-th layer feature reconstruction network is... The first reconstructed feature sequence is normalized by a summation and normalization layer if M equals 1, to obtain the second reconstructed feature sequence. Alternatively, if M is greater than 1, the first reconstructed feature sequence is weighted and summed by a summation and normalization layer, and the weighted summation result is normalized to obtain the second reconstructed feature sequence. The second reconstructed feature sequence is processed by other network layers in the i-th layer of the feature reconstruction network to obtain the third reconstructed feature sequence. Based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th layer of the feature reconstruction network, the feature difference score map of the first image is determined. Based on the feature difference score map of the first image, the abnormal pixels of the first image are determined. Since residual connections are not included above the attention network of the feature extraction network, the possibility of learning shortcuts can be reduced. At the same time, this application proposes to introduce a reference feature sequence in each layer of the feature reconstruction network. Since the reference feature sequence conforms to the normal image feature pattern, the similarity between abnormal features in the input feature sequence and the reference feature sequence is small, while the similarity between normal features in the input feature sequence and the reference feature sequence is large. This allows the first attention matrix corresponding to each feature in the input feature sequence to reflect its difference with different reference features, thereby reducing the possibility of rank collapse of the attention matrix and improving the reliability of anomaly detection.

[0216] Furthermore, the anomaly detection method provided in this application embodiment can be applied to both multi-category anomaly detection and single-category anomaly detection, and this application embodiment does not impose any limitations on this.

[0217] Figure 15 A flowchart illustrating a model training method provided in this application embodiment. This method can be executed by a training device, which may be... Figure 2 The training equipment 203 in the text is not limited to this, such as... Figure 15 As shown, the method may include:

[0218] S1510: Perform multi-scale feature processing on the second image through a feature extraction network to obtain a second multi-scale feature sequence;

[0219] S1520: Through each of the M attention networks in the i-th layer feature reconstruction network, the input feature sequence of the i-th layer feature reconstruction network is reconstructed based on the reference feature sequence in the i-th layer feature reconstruction network to obtain the fourth reconstructed feature sequence. Here, the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, and the input feature sequence of the 1st layer feature reconstruction network is the second multi-scale feature sequence, i = 1, 2... N, where N and M are both positive integers.

[0220] S1530: If M equals 1, then the fourth reconstructed feature sequence is normalized through a summation and normalization layer to obtain the fifth reconstructed feature sequence; or, if M is greater than 1, then the M fourth reconstructed feature sequences are weighted and summed through a summation and normalization layer, and the weighted summation result is normalized to obtain the fifth reconstructed feature sequence.

[0221] S1540: The fifth reconstructed feature sequence is processed by other network layers in the i-th layer feature reconstruction network to obtain the sixth reconstructed feature sequence;

[0222] S1550: Based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the Nth layer feature reconstruction network, calculate the loss of the target model composed of the feature extraction network and the Nth layer feature reconstruction network;

[0223] S1560: Based on the loss of the target model, train the target model and the reference feature sequence of each layer in the N-layer feature reconstruction network.

[0224] The following is a detailed explanation of S1510:

[0225] In some implementations, the execution device can capture a second image of the object to be detected using an image acquisition device. This image acquisition device can be an internal image acquisition unit of the execution device, such as a camera, or it can be a peripheral device of the execution device; this application embodiment does not impose any limitations on this.

[0226] It should be understood that the second image is a positive sample.

[0227] It should be understood that S1510 can be implemented in any of the following ways, but is not limited to:

[0228] In the first implementation, S1510 may include:

[0229] S1510-1A: Multi-scale feature extraction is performed on the second image using a pre-trained model in the feature extraction network to obtain the fifth feature map under multiple scales;

[0230] It should be understood that the explanation of the pre-trained model can be found above, and the embodiments of this application will not repeat it here.

[0231] It should be understood that the explanation of the fifth feature map under multiple scales can be referred to the explanation of the first feature map under multiple scales, and will not be repeated in the embodiments of this application.

[0232] S1510-2A: The fifth feature map at multiple scales is fused through the fusion layer in the feature extraction network to obtain the second fused feature map;

[0233] It should be understood that fusing the fifth feature maps at multiple scales can enhance image features, thereby improving the robustness of anomaly detection.

[0234] In some feasible implementations, the training device can perform average pooling operations on the fifth feature maps at multiple scales through a fusion layer to obtain multiple sixth feature maps at the same scale; and then merge the multiple sixth feature maps at the same scale along the channel dimension through the fusion layer to obtain a second fused feature map.

[0235] In some feasible implementations, the execution device can perform padding or cropping operations on the fifth feature maps at multiple scales through a fusion layer to obtain multiple sixth feature maps at the same scale; and then merge the multiple sixth feature maps at the same scale along the channel dimension through the fusion layer to obtain a second fused feature map.

[0236] It should be understood that the average pooling operation and feature map merging method can be referred to above, and will not be repeated in the embodiments of this application.

[0237] S1510-3A: The second fused feature map is shaped by the transformation layer in the feature extraction network to obtain the third multi-scale feature sequence, and the second multi-scale feature sequence is obtained based on the third multi-scale feature sequence.

[0238] It should be understood that the shape transformation method for the second fused feature map can refer to the shape transformation method for the first fused feature map, and this application embodiment will not elaborate further.

[0239] In some feasible implementations, the training device can perturb the third multi-scale feature sequence through a transformation layer to obtain the second multi-scale feature sequence.

[0240] In some feasible implementations, the training device can randomly sample a noise token sequence from a Gaussian distribution, sum the third-scale feature sequence with the noise token sequence to obtain a second multi-scale feature sequence, but is not limited to this.

[0241] Figure 16A schematic diagram of another feature extraction network provided in the embodiments of this application is shown below. Figure 16 As shown, the sixth feature map under the multi-scale is processed to form the third multi-scale feature sequence. Then, the third scale feature sequence is summed with the noise label sequence to obtain the second multi-scale feature sequence, where each box in the second multi-scale feature sequence represents a D-dimensional vector.

[0242] It should be understood that, by perturbing the third multi-scale feature sequence in the embodiments of this application, the output of the attention network in the feature reconstruction network includes the perturbed multi-scale feature sequence, i.e., the second multi-scale feature sequence, rather than directly including the third multi-scale feature sequence. This can reduce the possibility of learning shortcuts and thus improve the generalization ability and robustness of the target model.

[0243] Option 2 can be implemented, and S1510 may include:

[0244] S1510-1B: Input the size requirements that the second image and the third multi-scale feature sequence need to meet into the neural network model to obtain the third multi-scale feature sequence;

[0245] S1510-2B: Based on the third multi-scale feature sequence, the second multi-scale feature sequence is obtained.

[0246] It should be understood that the explanation of S1510-1B can be referred to the explanation of S310-1B, and the explanation of S1510-2B can be referred to the explanation of how to obtain the second multi-scale feature sequence based on the third multi-scale feature sequence described above. The embodiments of this application will not be repeated here.

[0247] The following is a detailed explanation of S1520:

[0248] In some implementations, the M attention networks include at least one of the following: a first attention network and a second attention network; correspondingly, the input feature sequence of the i-th layer feature reconstruction network is reconstructed based on the reference feature sequence in the i-th layer feature reconstruction network through each of the M attention networks in the i-th layer feature reconstruction network, to obtain a fourth reconstructed feature sequence, which includes at least one of the following:

[0249] 1. Using the first attention network, each feature in the input feature sequence of the i-th layer feature reconstruction network is converted into a third query vector and a third value vector, and each feature in the reference feature sequence of the i-th layer feature reconstruction network is converted into a third key vector; based on the third query vector and the third key vector, the fifth attention matrix corresponding to each feature in the input feature sequence of the i-th layer feature reconstruction network is calculated; based on the fifth attention matrix and the third value vector, the fourth reconstructed feature sequence is calculated.

[0250] 2. Using the second attention network, each feature in the input feature sequence of the i-th layer feature reconstruction network is converted into a fourth query vector, and each feature in the reference feature sequence of the i-th layer feature reconstruction network is converted into a fourth key vector and a fourth value vector; based on the fourth query vector and the fourth key vector, the sixth attention matrix corresponding to each feature in the input feature sequence of the i-th layer feature reconstruction network is calculated; based on the sixth attention matrix and the fourth value vector, the fourth reconstructed feature sequence is calculated.

[0251] The following section explains the case of obtaining the fourth reconstructed feature sequence through the first attention network:

[0252] It should be understood that since the third key vector is derived from the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, when calculating the fifth attention matrix based on the third key vector and the third query vector, the abnormal features in the input feature sequence, i.e. the features corresponding to the abnormal regions, have a low similarity to the third key vector. Therefore, the fourth reconstructed feature sequence usually does not retain the abnormal image feature pattern, but most likely retains the normal image feature pattern, which makes the difference between the input feature sequence and the reconstructed feature sequence larger, thus facilitating anomaly detection.

[0253] Suppose that each feature in the input feature sequence of the i-th layer feature reconstruction network is called the third feature. Then, the execution device can obtain the fifth attention matrix corresponding to the third feature through the first attention network in any of the following possible ways, but is not limited to:

[0254] In some implementations, the execution device uses a first attention network to perform a scaled dot product between the third query vector corresponding to the third feature and the transpose of each third key vector, and then performs normalization to obtain the fifth attention matrix corresponding to the third feature.

[0255] In some implementations, the execution device uses a first attention network to perform a scaled dot product between the third query vector corresponding to the third feature and the transpose of each third key vector to obtain a seventh attention matrix corresponding to the third feature; and calculates the sum of the seventh attention matrix corresponding to the third feature and the third mask matrix corresponding to the third feature, and performs normalization processing to obtain a fifth attention matrix corresponding to the third feature; wherein, the value of the position corresponding to the third feature and the surrounding positions in the third mask matrix are negative infinity, and the value of the remaining positions is 0.

[0256] The third mask matrix, also known as the neighbor masking attention matrix or attention graph, is used to mask the attention values ​​of the third feature and its neighboring features in the seventh attention matrix corresponding to the third feature.

[0257] In some possible implementations, the neighbor features of the third feature may include at least one of the following, but not limited to: the neighbor features directly above the third feature, the neighbor features directly below the third feature, the neighbor features to the left of the third feature, the neighbor features to the right of the third feature, the neighbor features to the top left of the third feature, the neighbor features to the bottom left of the third feature, the neighbor features to the top right of the third feature, and the neighbor features to the bottom right of the third feature.

[0258] It should be understood that the surrounding positions of the third feature refer to the corresponding positions of the neighboring features of the third feature in the third mask matrix.

[0259] It should be understood that the problem of learning shortcuts is mainly caused by the fact that the network output directly includes the network input. Based on this, the embodiments of this application propose to mask the attention values ​​corresponding to the third feature and its neighboring features in the seventh attention matrix corresponding to the third feature, so that when reconstructing the third feature, the third value vector corresponding to the third feature and its neighboring features can be ignored, thereby preventing the reconstructed feature from including the third feature and its neighboring features, reducing the possibility of learning shortcuts, and thus improving the robustness of the anomaly detection method.

[0260] The following section elaborates on the case of obtaining the fourth reconstructed feature sequence through the second attention network:

[0261] It should be understood that since the fourth key vector is derived from features in the reference feature sequence, and the reference feature sequence conforms to normal image feature patterns, when calculating the sixth attention matrix based on the fourth key vector and the fourth query vector, the anomalous features in the input feature sequence—that is, the features corresponding to the anomalous regions—have low similarity to the fourth key vector. Therefore, the fourth reconstructed feature sequence usually does not retain anomalous image feature patterns, but rather retains normal image feature patterns, resulting in a larger difference between the input feature sequence and the reconstructed feature sequence, which is beneficial for anomaly detection. Furthermore, since the fourth value vector is also derived from features in the reference feature sequence, and the reference feature sequence conforms to normal image feature patterns, the generated fourth reconstructed feature sequence based on the fourth value vector is more likely to retain normal image feature patterns, further increasing the difference between the input feature sequence and the reconstructed feature sequence, and thus further improving the robustness of the anomaly detection method.

[0262] Suppose that each feature in the input feature sequence of the i-th layer feature reconstruction network is called the fourth feature. Then, the execution device can obtain the sixth attention matrix corresponding to the fourth feature through the second attention network in any of the following possible ways, but is not limited to:

[0263] In some implementations, the execution device uses a second attention network to perform a scaled dot product between the fourth query vector corresponding to the fourth feature and the transpose of each fourth key vector, and then performs normalization to obtain the sixth attention matrix corresponding to the fourth feature.

[0264] In some implementations, the execution device uses a second attention network to perform a scaled dot product between the fourth query vector corresponding to the fourth feature and the transpose of each fourth key vector to obtain the eighth attention matrix corresponding to the fourth feature; and calculates the sum of the eighth attention matrix corresponding to the fourth feature and the fourth mask matrix corresponding to the fourth feature, and performs normalization processing to obtain the sixth attention matrix corresponding to the fourth feature; wherein, the value of the position corresponding to the fourth feature and the surrounding positions in the fourth mask matrix is ​​0, and the value of the remaining positions is negative infinity.

[0265] The fourth mask matrix is ​​used to mask the attention values ​​of the other features in the eighth attention matrix corresponding to the fourth feature, excluding the fourth feature and its neighboring features.

[0266] In some possible implementations, the neighbor features of the fourth feature may include at least one of the following, but not limited to: the neighbor features directly above the fourth feature, the neighbor features directly below the fourth feature, the neighbor features to the left of the fourth feature, the neighbor features to the right of the fourth feature, the neighbor features to the top left of the fourth feature, the neighbor features to the bottom left of the fourth feature, the neighbor features to the top right of the fourth feature, and the neighbor features to the bottom right of the fourth feature.

[0267] It should be understood that the surrounding positions of the fourth feature refer to the corresponding positions of the neighboring features of the fourth feature in the fourth mask matrix.

[0268] It should be understood that since the fourth value vector is the result of feature transformation in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, this application embodiment proposes to mask the attention values ​​corresponding to other features in the eighth attention matrix corresponding to the fourth feature, except for the fourth feature and its neighboring features, so that when reconstructing the fourth feature, the fourth value vector corresponding to the fourth feature and its neighboring features can be referenced in a key way, thereby further improving the robustness of the anomaly detection method.

[0269] It should be understood that, when the first attention network and the second attention network are combined, or when the first attention network based on the MLKA mechanism and the first attention network based on the LCA mechanism are combined, for each feature in the input feature sequence, the first attention network can reconstruct the feature to obtain a reconstructed feature. Similarly, the second attention network can also reconstruct the feature to obtain a reconstructed feature. The summation and normalization layer can perform a weighted summation of the two reconstructed features and then perform normalization to obtain a reconstructed feature corresponding to that feature.

[0270] It should be noted that the third feature mentioned above can be every feature of the input feature sequence, and the fourth feature mentioned above can also be every feature of the input feature sequence. The third feature and the fourth feature can be the same feature or different features. It is worth mentioning that when the first attention network is combined with the second attention network, or when the first attention network based on the MLKA mechanism is combined with the first attention network based on the LCA mechanism, the third feature and the fourth feature refer to the same feature.

[0271] The S330 will be described in detail below:

[0272] In some implementations, if M equals 1, the execution device normalizes the fourth reconstructed feature sequence through a summation and normalization layer to obtain the fifth reconstructed feature sequence.

[0273] In some implementations, if M is greater than 1, the execution device performs a weighted summation on the M fourth reconstructed feature sequences through a summation and normalization layer to obtain a weighted summation result; and then normalizes the weighted summation result to obtain the fifth reconstructed feature sequence.

[0274] The following is a detailed explanation of S1540:

[0275] In some implementations, the other network layers in the i-th layer feature reconstruction network include a feedforward network layer and another summing and normalization layer. Based on this, the training device processes the fifth reconstructed feature sequence through the other network layers in the i-th layer feature reconstruction network to obtain the sixth reconstructed feature sequence. This process includes: the training device performing linear and nonlinear transformations on the fifth reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain a second transformation result; then summing the second transformation result and the fifth reconstructed feature sequence through the summing and normalization layer; and finally normalizing the summing result to obtain the sixth reconstructed feature sequence.

[0276] In some implementations, the other network layers in the i-th layer feature reconstruction network include a feedforward network layer and another summing and normalization layer. Based on this, the training device processes the fifth reconstructed feature sequence through the other network layers in the i-th layer feature reconstruction network to obtain the sixth reconstructed feature sequence. This includes the training device performing linear and nonlinear transformations on the fifth reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain the sixth reconstructed feature sequence.

[0277] It should be understood that the embodiments of this application do not limit other network layers in the i-th layer feature reconstruction network, nor do they limit the corresponding processing of other network layers.

[0278] The S1550 will be explained below:

[0279] In some implementations, the training device can restore the sixth reconstructed feature sequence output by the Nth layer feature reconstruction network to the seventh feature map at multiple scales; restore the second multi-scale feature sequence to the eighth feature map at multiple scales; and calculate the loss of the target model based on the seventh and eighth feature maps at multiple scales.

[0280] It should be understood that the above text provides information on restoring the reconstructed feature sequence into a feature map at multiple scales, and this application will not repeat the details in the embodiments.

[0281] In some implementations, the training device can calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale; and sum the losses of the target model at each scale to obtain the total loss of the target model.

[0282] In some feasible approaches, the training device can utilize cosine similarity (Cos) to calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale.

[0283] For example, suppose the cosine loss between the j-th seventh feature map and the j-th eighth feature map is denoted as... Let the j-th seventh feature map be denoted as Let the j-th eighth feature map be denoted as Let H be the height and width of the seventh and eighth feature maps, respectively. j and W j Let the total loss be denoted as L. Then, the total loss L can be calculated using the following formula:

[0284]

[0285] in, express and The corresponding element-wise dot product is then summed.

[0286] In some feasible approaches, the training device can utilize mean squared error (MSE) to calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale.

[0287] For example, suppose the j-th seventh feature map is denoted as Let the j-th eighth feature map be denoted as Let H be the height and width of the seventh and eighth feature maps, respectively. j and W j Let the total loss be denoted as L. Then, the total loss L can be calculated using the following formula:

[0288]

[0289] In some feasible approaches, the training device can use cosine similarity (Cos) and mean squared error (MSE) to calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale.

[0290] For example, suppose the cosine loss between the j-th seventh feature map and the j-th eighth feature map is denoted as... Let the j-th seventh feature map be denoted as Let the j-th eighth feature map be denoted as Let H be the height and width of the seventh and eighth feature maps, respectively. j and W j Let the total loss be denoted as L. Then, the total loss L can be calculated using the following formula:

[0291]

[0292] The explanations of the symbols in these two formulas can be found above, and will not be repeated in this embodiment.

[0293] Figure 17 This is a schematic diagram illustrating the calculation of the target model loss provided in an embodiment of this application, as shown below. Figure 17 As shown, the training device can use mean squared error (MSE) and cosine similarity (Cos) to calculate the loss of the seventh feature map and the eighth feature map under multiple scales, and obtain the total loss of the target model.

[0294] It should be understood that the embodiments of this application do not limit the calculation method of the target model loss.

[0295] The following is an explanation of S1560:

[0296] In some implementations, the training device can adjust the parameters of the target model based on the total loss of the target model until the training iterations reach a preset number or the total loss reaches a minimum value, at which point training stops.

[0297] It should be noted that before training the target model, an initial value can be assigned to the reference feature sequence of each layer in the N-layer feature reconstruction network. When training the target model, the reference feature sequence of each layer can be adjusted at the same time. Since this application proposes not to include residual connections above the attention network of the feature extraction network, the possibility of learning shortcuts can be reduced. Based on this, while training the target model, the reference feature sequence can learn the normal image feature pattern.

[0298] The following will combine Figure 18 The anomaly detection method provided in the embodiments of this application will be described by way of example:

[0299] Let the model consisting of the feature extraction network and the N-layer feature reconstruction network be called the target model. Figure 18 A schematic diagram of another target model provided in the embodiments of this application, as shown below. Figure 18As shown, the training device can perform multi-scale feature extraction on the image using a pre-trained model in the feature extraction network to obtain multi-scale feature maps. Further, the multi-scale feature maps are processed to obtain a third multi-scale feature sequence, and the third multi-scale feature sequence and the noise label sequence are summed to obtain a second multi-scale feature sequence. Each feature extraction network in the N-layer feature reconstruction network includes: a first attention network and a second attention network, a summation and normalization layer, a feedforward network, and another summation and normalization layer. Specifically, the training device can use the first attention network to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a third query vector and a third value vector, and convert each feature in the reference feature sequence of the i-th layer feature reconstruction network into a third key vector. For each feature in the input feature sequence, the training device uses a first attention network to perform a scaled dot product between the third query vector corresponding to that feature and the transpose of each third key vector, obtaining the seventh attention matrix corresponding to that feature. The sum of the seventh attention matrix and the third mask matrix corresponding to that feature is then calculated and normalized to obtain the fifth attention matrix corresponding to that feature. Further, the training device uses the first attention network to calculate the product of the fifth attention matrix corresponding to that feature and the matrix formed by all the third value vectors, obtaining the reconstructed feature corresponding to that feature. The reconstructed features corresponding to each feature in the input feature sequence constitute the fourth reconstructed feature sequence. Similarly, the training device uses a second attention network to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a fourth query vector, and converts each feature in the reference feature sequence of the i-th layer feature reconstruction network into a fourth key vector and a fourth value vector. For each feature in the input feature sequence, the training device uses a second attention network to perform a scaled dot product between the fourth query vector corresponding to that feature and the transpose of each fourth key vector, obtaining the eighth attention matrix corresponding to that feature. The sum of the eighth attention matrix and the fourth mask matrix corresponding to that feature is calculated and normalized to obtain the sixth attention matrix. Further, the training device uses the second attention network to calculate the product of the sixth attention matrix and the matrix formed by all the fourth value vectors, obtaining the reconstructed feature corresponding to that feature. The reconstructed features corresponding to each feature in the input feature sequence constitute the fourth reconstructed feature sequence. Further still, a summation and normalization layer performs a weighted summation of the two fourth reconstructed feature sequences and normalizes it to obtain the fifth reconstructed feature sequence. Finally, the training device uses a feedforward network layer to perform linear and nonlinear transformations on the fifth reconstructed feature sequence to obtain the transformation result. Then, the summation and normalization layer sums the transformation result and the fifth reconstructed feature sequence, and normalizes the summation result to obtain the sixth reconstructed feature sequence.

[0300] This application provides a model training method, including: performing multi-scale feature processing on a second image through a feature extraction network to obtain a second multi-scale feature sequence; reconstructing the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence in the i-th layer feature reconstruction network through each of the M attention networks in the i-th layer feature reconstruction network to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, the input feature sequence of the 1-th layer feature reconstruction network is the second multi-scale feature sequence, i = 1, 2...N, where N and M are both positive integers; if M equals 1, then by summation and normalization... The first layer normalizes the fourth reconstructed feature sequence to obtain the fifth reconstructed feature sequence; or, if M is greater than 1, the fifth reconstructed feature sequence is obtained by weighted summation of the M fourth reconstructed feature sequences through a summation and normalization layer, followed by normalization of the weighted summation result. The fifth reconstructed feature sequence is then processed by other network layers in the i-th layer of the feature reconstruction network to obtain the sixth reconstructed feature sequence. Based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th layer of the feature reconstruction network, the loss of the target model composed of the feature extraction network and the N-layer feature reconstruction network is calculated. Based on the loss of the target model, the reference feature sequences of the target model and each layer of the N-layer feature reconstruction network are trained. Since residual connections are not included above the attention network of the feature extraction network, the possibility of learning shortcuts can be reduced, allowing the reference feature sequences of each layer and the target model to learn normal image feature patterns. This makes it highly probable that the input feature sequence can be reconstructed into a normal feature sequence when feature reconstruction is performed based on the target model and the reference feature sequence, thus making the difference between the input feature sequence with abnormal features and the reconstructed feature sequence larger, which is beneficial for anomaly detection.

[0301] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0302] It should also be understood that, in the various method embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0303] The methods provided in the embodiments of this application have been described above. The anomaly detection device and model training device provided in the embodiments of this application will be described below.

[0304] Figure 19 This is a schematic diagram of an anomaly detection device 1900 provided in an embodiment of this application. The device 1900 includes:

[0305] The multi-scale feature processing module 1910 is used to perform multi-scale feature processing on the first image through a feature extraction network to obtain a first multi-scale feature sequence.

[0306] The first feature reconstruction module 1920 is used to reconstruct the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence in the i-th layer feature reconstruction network through each of the M attention networks in the i-th layer feature reconstruction network, to obtain the first reconstructed feature sequence. Here, the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, and the input feature sequence of the 1-th layer feature reconstruction network is the first multi-scale feature sequence, i = 1, 2...N, where N and M are both positive integers, and the reference feature sequence in the i-th layer feature reconstruction network conforms to the normal image feature pattern.

[0307] The second feature reconstruction module 1930 is used to normalize the first reconstructed feature sequence through a summation and normalization layer if M equals 1, to obtain the second reconstructed feature sequence; or, if M is greater than 1, to perform weighted summation on the M first reconstructed feature sequences through a summation and normalization layer, and to normalize the weighted summation result to obtain the second reconstructed feature sequence.

[0308] The third feature reconstruction module 1940 is used to process the second reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain the third reconstructed feature sequence;

[0309] The first determining module 1950 is used to determine the feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the Nth layer feature reconstruction network.

[0310] The second determining module 1960 is used to determine abnormal pixels in the first image based on the feature difference score map of the first image.

[0311] In some implementations, the M attention networks include at least one of the following: a first attention network and a second attention network; correspondingly, the first feature reconstruction module 1920 is used to perform at least one of the following:

[0312] Through the first attention network, each feature in the input feature sequence of the i-th layer feature reconstruction network is converted into a first query vector and a first value vector, and each feature in the reference feature sequence of the i-th layer feature reconstruction network is converted into a first key vector; based on the first query vector and the first key vector, the first attention matrix corresponding to each feature in the input feature sequence of the i-th layer feature reconstruction network is calculated; based on the first attention matrix and the first value vector, the first reconstructed feature sequence is calculated.

[0313] The second attention network is used to convert each feature in the input feature sequence of the i-th layer feature reconstruction network into a second query vector, and convert each feature in the reference feature sequence of the i-th layer feature reconstruction network into a second key vector and a second value vector. The second attention matrix corresponding to each feature in the input feature sequence of the i-th layer feature reconstruction network is calculated based on the second query vector and the second key vector. The first reconstructed feature sequence is calculated based on the second attention matrix and the second value vector.

[0314] In some implementations, the first feature reconstruction module 1920 is specifically used to: perform a scaled dot product on the first query vector corresponding to the first feature and the transpose of each first key vector to obtain the third attention matrix corresponding to the first feature; and calculate the sum of the third attention matrix corresponding to the first feature and the first mask matrix corresponding to the first feature, and perform normalization processing to obtain the first attention matrix corresponding to the first feature; wherein, the value of the position corresponding to the first feature and the surrounding positions in the first mask matrix is ​​negative infinity, and the value of the remaining positions is 0.

[0315] In some implementations, the first feature reconstruction module 1920 is specifically used to: perform a scaled dot product on the second query vector corresponding to the second feature and the transpose of each second key vector to obtain the fourth attention matrix corresponding to the second feature; wherein, the second feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; calculate the sum of the fourth attention matrix corresponding to the second feature and the second mask matrix corresponding to the second feature, and perform normalization processing to obtain the second attention matrix corresponding to the second feature; wherein, the value of the position corresponding to the second feature and the surrounding positions in the second mask matrix is ​​0, and the value of the remaining positions is negative infinity.

[0316] In some implementations, when the first reconstructed feature sequence output by the first attention network and the first reconstructed feature sequence output by the second attention network are weighted and summed through a summation and normalization layer, the weight corresponding to the first reconstructed feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstructed feature sequence output by the second attention network is a hyperparameter greater than 1.

[0317] In some implementations, the multi-scale feature processing module 1910 is specifically used to: extract multi-scale features from the first image using a pre-trained model in the feature extraction network to obtain a first feature map at multiple scales; fuse the first feature map at multiple scales using a fusion layer in the feature extraction network to obtain a first fused feature map; and transform the shape of the first fused feature map using a transformation layer in the feature extraction network to obtain a first multi-scale feature sequence.

[0318] In some implementations, the multi-scale feature processing module 1910 is specifically used to: perform average pooling operations on the first feature maps at multiple scales through a fusion layer to obtain multiple second feature maps at the same scale; and merge the multiple second feature maps at the same scale along the channel dimension through a fusion layer to obtain a first fused feature map.

[0319] In some possible implementations, the first determining module 1950 is specifically used to: restore the third reconstructed feature sequence output by the Nth layer feature reconstruction network to a third feature map at multiple scales; restore the first multi-scale feature sequence to a fourth feature map at multiple scales; and calculate the feature difference score between the third feature map at multiple scales and the fourth feature map at multiple scales to obtain the feature difference score map of the first image.

[0320] In some implementations, the second determining module 1960 is specifically used to: determine pixels in the feature difference score map that have a score higher than a preset score as abnormal pixels in the first image.

[0321] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 19 The device 1900 shown can perform Figure 3 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in the apparatus 1900 are respectively for implementing Figure 3 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0322] The apparatus 1900 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0323] Figure 20 A schematic diagram of a model training device 2000 provided in an embodiment of this application is shown below. Figure 20 As shown, the device 2000 includes:

[0324] The multi-scale feature processing module 2010 is used to perform multi-scale feature processing on the second image through a feature extraction network to obtain a second multi-scale feature sequence;

[0325] The first feature reconstruction module 2020 is used to reconstruct the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence in the i-th layer feature reconstruction network through each of the M attention networks in the i-th layer feature reconstruction network, to obtain the fourth reconstructed feature sequence. Here, the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, and the input feature sequence of the 1-th layer feature reconstruction network is the second multi-scale feature sequence, i = 1, 2...N, where N and M are both positive integers.

[0326] The second feature reconstruction module 2030 is used to normalize the fourth reconstructed feature sequence through a summation and normalization layer if M equals 1, to obtain the fifth reconstructed feature sequence; or, if M is greater than 1, to perform weighted summation on the M fourth reconstructed feature sequences through a summation and normalization layer, and to normalize the weighted summation result to obtain the fifth reconstructed feature sequence.

[0327] The third feature reconstruction module 2040 is used to process the fifth reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain the sixth reconstructed feature sequence;

[0328] The loss calculation module 2050 is used to calculate the loss of the target model composed of the feature extraction network and the N-layer feature reconstruction network based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-layer feature reconstruction network.

[0329] Training module 2060 is used to train the target model and the reference feature sequence for each layer in the N-layer feature reconstruction network based on the loss of the target model.

[0330] In some implementations, the multi-scale feature processing module 2010 is specifically used to: extract multi-scale features from the second image using a pre-trained model in the feature extraction network to obtain a fifth feature map at multiple scales; fuse the fifth feature map at multiple scales using a fusion layer in the feature extraction network to obtain a second fused feature map; transform the shape of the second fused feature map using a transformation layer in the feature extraction network to obtain a third multi-scale feature sequence, and obtain a second multi-scale feature sequence based on the third multi-scale feature sequence.

[0331] In some implementations, the multi-scale feature processing module 2010 is specifically used to: perform average pooling operations on the fifth feature maps at multiple scales through the fusion layer to obtain multiple sixth feature maps at the same scale; and merge the multiple sixth feature maps at the same scale along the channel dimension through the fusion layer to obtain a second fused feature map.

[0332] In some implementations, the multi-scale feature processing module 2010 is specifically used to: perturb the third multi-scale feature sequence to obtain the second multi-scale feature sequence.

[0333] In some possible implementations, the loss calculation module 2050 is specifically used to: restore the sixth reconstructed feature sequence output by the Nth layer feature reconstruction network to the seventh feature map under multiple scales; restore the second multi-scale feature sequence to the eighth feature map under multiple scales; and calculate the loss of the target model based on the seventh feature map and the eighth feature map under multiple scales.

[0334] In some possible implementations, the loss calculation module 2050 is specifically used to: calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale; and sum the losses of the target model at each scale to obtain the total loss of the target model.

[0335] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 20 The device 2000 shown can perform Figure 15 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in the device 2000 are respectively for implementing Figure 15For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0336] The apparatus 2000 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0337] Figure 21 This is a schematic block diagram of the electronic device 2100 provided in an embodiment of this application. The electronic device 2100 may be an execution device or a training device, but is not limited thereto. Figure 21 As shown, the electronic device 2100 may include:

[0338] The system includes a memory 2110 and a processor 2120. The memory 2110 stores a computer program 2130 and transfers the computer program 2130 to the processor 2120. In other words, the processor 2120 can retrieve and run the computer program 2130 from the memory 2110 to implement the methods described in the embodiments of this application.

[0339] For example, the processor 2120 can be used to execute the steps in the above method according to the instructions in the computer program 2130.

[0340] In some embodiments of this application, the processor 2120 may include, but is not limited to:

[0341] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0342] In some embodiments of this application, the memory 2110 includes, but is not limited to:

[0343] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct RAM (DR RAM).

[0344] In some embodiments of this application, the computer program 2130 may be divided into one or more modules, which are stored in the memory 2110 and executed by the processor 2120 to complete the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 2130 in the electronic device.

[0345] like Figure 21 As shown, the electronic device 2100 may further include:

[0346] Transceiver 2140, which can be connected to processor 2120 or memory 2110.

[0347] The processor 2120 can control the transceiver 2140 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 2140 may include a transmitter and a receiver. The transceiver 2140 may further include antennas, and the number of antennas may be one or more.

[0348] It should be understood that the various components in the electronic device 2100 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0349] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0350] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0351] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0352] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0353] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0354] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0355] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An anomaly detection method characterized by, The method comprises the following steps: performing multi-scale feature processing on the first image through a feature extraction network to obtain a first multi-scale feature sequence; reconstructing, through each of the M attention networks in the i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence in the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the first multi-scale feature sequence, i=1, 2, …, N, N and M are positive integers, and the reference feature sequence in the i-th layer feature reconstruction network conforms to a normal image feature mode; if M is equal to 1, performing normalization processing on the first reconstructed feature sequence through a summing and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, performing weighted summing on the M first reconstructed feature sequences through the summing and normalization layer, and performing normalization processing on the weighted summing result to obtain the second reconstructed feature sequence; processing the second reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence; determining a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by an N-th layer feature reconstruction network; determining an abnormal pixel point of the first image based on the feature difference score map of the first image.

2. The method of claim 1, wherein, The M attention networks comprise at least one of the following: a first attention network and a second attention network; The reconstruction, through each of the M attention networks in the i-th layer feature reconstruction network, of the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence in the i-th layer feature reconstruction network to obtain the first reconstructed feature sequence comprises at least one of the following: Through the first attention network, each feature in the input feature sequence in the i-th layer feature reconstruction network is converted into a first query vector and a first value vector, and each feature in the reference feature sequence in the i-th layer feature reconstruction network is converted into a first key vector; a first attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the first query vector and the first key vector; the first reconstructed feature sequence is calculated based on the first attention matrix and the first value vector; Through the second attention network, each feature in the input feature sequence in the i-th layer feature reconstruction network is converted into a second query vector, and each feature in the reference feature sequence in the i-th layer feature reconstruction network is converted into a second key vector and a second value vector; a second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the second query vector and the second key vector; the first reconstructed feature sequence is calculated based on the second attention matrix and the second value vector.

3. The method of claim 2, wherein, The first attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the first query vector and the first key vector, including: The first feature corresponding to the first query vector is scaled dot product with the transpose of each first key vector to obtain the third attention matrix corresponding to the first feature; wherein the first feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; The sum of the third attention matrix corresponding to the first feature and the first mask matrix corresponding to the first feature is calculated, and normalized to obtain the first attention matrix corresponding to the first feature; Wherein the values of the first feature corresponding position and the surrounding position in the first mask matrix are negative infinity, and the values of the remaining positions are 0.

4. The method of claim 2, wherein, The second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the second query vector and the second key vector, including: The second feature corresponding to the second query vector is scaled dot product with the transpose of each second key vector to obtain the fourth attention matrix corresponding to the second feature; wherein the second feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; The sum of the fourth attention matrix corresponding to the second feature and the second mask matrix corresponding to the second feature is calculated, and normalized to obtain the second attention matrix corresponding to the second feature; Wherein the values of the second feature corresponding position and the surrounding position in the second mask matrix are 0, and the values of the remaining positions are negative infinity.

5. The method according to any one of claims 2-4, characterized in that, When the first reconstruction feature sequence output by the first attention network and the first reconstruction feature sequence output by the second attention network are weighted and summed by the sum and normalization layer, the weight corresponding to the first reconstruction feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstruction feature sequence output by the second attention network is a hyperparameter greater than 1.

6. The method according to any one of claims 1 to 4, characterized in that, The first multi-scale feature sequence is obtained by performing multi-scale feature processing on the first image through the feature extraction network, including: The first feature map under multiple scales is obtained by performing multi-scale feature extraction on the first image through the pre-trained model in the feature extraction network; The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network; The first multi-scale feature sequence is obtained by shape conversion of the first fusion feature map through the conversion layer in the feature extraction network.

7. The method of claim 6, wherein, The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network, including: The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network, including: The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network, including:

8. A model training method, comprising: Including: The second image is processed by a feature extraction network to obtain a second multi-scale feature sequence; Each of the M attention networks in the i-th feature reconstruction network reconstructs the input feature sequence of the i-th feature reconstruction network based on the reference feature sequence in the i-th feature reconstruction network, to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is the output feature sequence of the (i-1)-th feature reconstruction network, the input feature sequence of the first feature reconstruction network is the second multi-scale feature sequence, i=1, 2, …, N, and N and M are positive integers; If M is equal to 1, the fourth reconstructed feature sequence is normalized by a sum and normalization layer to obtain a fifth reconstructed feature sequence; or if M is greater than 1, the M fourth reconstructed feature sequences are weighted summed by the sum and normalization layer, and the weighted sum result is normalized to obtain the fifth reconstructed feature sequence; The fifth reconstructed feature sequence is processed by other network layers in the i-th feature reconstruction network to obtain a sixth reconstructed feature sequence; Loss of a target model composed of the feature extraction network and the N feature reconstruction networks is calculated based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th feature reconstruction network; The target model and the reference feature sequence of each layer in the N feature reconstruction networks are trained based on the loss of the target model.

9. The method of claim 8, wherein, The second image is processed by a feature extraction network to obtain a second multi-scale feature sequence, including: The second image is processed by a pre-trained model in the feature extraction network to obtain a fifth feature map under multiple scales; The fifth feature map under multiple scales is fused by a fusion layer in the feature extraction network to obtain a second fused feature map; The second fused feature map is shape-converted by a conversion layer in the feature extraction network to obtain a third multi-scale feature sequence, and the second multi-scale feature sequence is obtained based on the third multi-scale feature sequence.

10. The method of claim 9, wherein, The second multi-scale feature sequence is obtained based on the third multi-scale feature sequence, including: The third multi-scale feature sequence is subjected to a perturbation operation to obtain the second multi-scale feature sequence.

11. An abnormality detection device characterized by comprising: Including: A multi-scale feature processing module is configured to process a first image by a feature extraction network to obtain a first multi-scale feature sequence; The first feature reconstruction module is configured to reconstruct, by each of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on a reference feature sequence in the i-th feature reconstruction network, to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i=1, 2, …, N, N and M are positive integers, and the reference feature sequence in the i-th feature reconstruction network conforms to a normal image feature mode. The second feature reconstruction module is configured to, if M is equal to 1, normalize the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, perform weighted summation on the M first reconstructed feature sequences by the sum and normalization layer, and perform normalization on the weighted summation result to obtain the second reconstructed feature sequence. The third feature reconstruction module is configured to process the second reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a third reconstructed feature sequence. The first determination module is configured to determine a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th feature reconstruction network. The second determination module is configured to determine an abnormal pixel point of the first image based on the feature difference score map of the first image.

12. A model training apparatus, comprising: The multi-scale feature processing module is configured to perform multi-scale feature processing on a second image by a feature extraction network to obtain a second multi-scale feature sequence. The first feature reconstruction module is configured to reconstruct, by each of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on a reference feature sequence in the i-th feature reconstruction network, to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i=1, 2, …, N, N and M are positive integers, and the reference feature sequence in the i-th feature reconstruction network conforms to a normal image feature mode. The second feature reconstruction module is configured to, if M is equal to 1, normalize the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, perform weighted summation on the M first reconstructed feature sequences by the sum and normalization layer, and perform normalization on the weighted summation result to obtain the second reconstructed feature sequence. The third feature reconstruction module is configured to process the second reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a third reconstructed feature sequence. The loss calculation module is configured to calculate a loss of a target model composed of the feature extraction network and the N feature reconstruction networks based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th feature reconstruction network. ​ a training module configured to train the target model and the reference feature sequence of each layer in the N-layer feature reconstruction network based on a loss of the target model.

13. An electronic device, comprising: comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the method of any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, a computer program product for storing a computer program, which causes a computer to execute the method of any one of claims 1 to 10.

15. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the method of any one of claims 1 to 10.

Images