Anomaly detection method and apparatus, model training method and apparatus, medium, and program product

Abstract

The present application relates to AI technology, and provides an anomaly detection method and apparatus, a model training method and apparatus, a medium, and a program product. The anomaly detection method comprises: performing multi-scale feature processing on a first image by means of a feature extraction network to obtain a first multi-scale feature sequence; by means of attention networks in an ith-layer feature reconstruction network, reconstructing an input feature sequence of the layer on the basis of a reference feature sequence of the layer, so as to obtain a first reconstructed feature sequence; by means of a summation and normalization layer in the ith-layer feature reconstruction network, processing the first reconstructed feature sequence to obtain a second reconstructed feature sequence; by means of other network layers in the ith-layer feature reconstruction network, processing the second reconstructed feature sequence to obtain a third reconstructed feature sequence; determining a feature difference score map of the first image on the basis of the first multi-scale feature sequence and the final third reconstructed feature sequence; and determining abnormal pixels on the basis of the score map. Thus, the reliability of anomaly detection can be improved.

Description

Abnormality detection method, model training manner, device, medium and program product

[0001] Priority information

[0002] The present application claims priority from the Chinese patent application No. 2024112664327 entitled "Abnormality detection method, model training manner, device, medium and program product" and filed with the China Patent Office on September 10, 2024, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] Embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to an abnormality detection method, a model training manner, a device, a medium and a program product. BACKGROUND

[0004] Abnormality detection technology is an important technology in industrial AI quality inspection, and the purpose is to find abnormal defects on industrial parts.

[0005] The current abnormality detection method includes: an electronic device performs multi-scale feature processing on a to-be-detected image to obtain a multi-scale feature sequence; and reconstructs the multi-scale feature sequence through a feature reconstruction network to obtain a reconstructed feature sequence; and finally performs abnormality detection based on the multi-scale feature sequence and the reconstructed feature sequence.

[0006] Among them, the current feature reconstruction network is a Transfomer structure, and this structure has a learning shortcut problem, in other words, this structure is not easy to learn the normal image feature mode, so that for the actual abnormal image, it is not easy to reconstruct the feature sequence conforming to the normal image feature mode through the feature reconstruction network of this structure, and further leads to the problem of low reliability of abnormality detection based on the difference between the multi-scale feature sequence and the reconstructed feature sequence. SUMMARY

[0007] The present application provides an abnormality detection method, a model training manner, a device, a medium and a program product, thereby the reliability of abnormality detection can be improved.

[0008] In a first aspect, an embodiment of the present application provides an anomaly detection method, comprising: performing multi-scale feature processing on a first image by a feature extraction network to obtain a first multi-scale feature sequence; reconstructing, by each of M attention networks in an i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence input into the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the first multi-scale feature sequence, i=2...N, N and M are positive integers, and the reference feature sequence conforms to a normal image feature mode; if M is equal to 1, performing normalization processing on the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, performing weighted summation on the M first reconstructed feature sequences by the sum and normalization layer, and performing normalization processing on a weighted summation result to obtain the second reconstructed feature sequence; processing the second reconstructed feature sequence by other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence; determining a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by an N-th layer feature reconstruction network; and determining an abnormal pixel point in the first image based on the feature difference score map of the first image.

[0009] In a second aspect, an embodiment of the present application provides a model training method, comprising: performing multi-scale feature processing on a second image by a feature extraction network to obtain a second multi-scale feature sequence; reconstructing, by each of M attention networks in an i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence input into the i-th layer feature reconstruction network to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the second multi-scale feature sequence, i=2...N, N and M are positive integers; if M is equal to 1, performing normalization processing on the fourth reconstructed feature sequence by a sum and normalization layer to obtain a fifth reconstructed feature sequence; or if M is greater than 1, performing weighted summation on the M fourth reconstructed feature sequences by the sum and normalization layer, and performing normalization processing on a weighted summation result to obtain the fifth reconstructed feature sequence; processing the fifth reconstructed feature sequence by other network layers in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence; calculating a loss of a target model composed of the feature extraction network and N layer feature reconstruction networks based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th layer feature reconstruction network; and training the target model and the reference feature sequence of each layer in the N layer feature reconstruction network based on the loss of the target model.

[0010] In a third aspect, an anomaly detection apparatus is provided. The anomaly detection apparatus comprises: a multi-scale feature processing module configured to perform multi-scale feature processing on a first image by a feature extraction network to obtain a first multi-scale feature sequence;

[0011] a first feature reconstruction module configured to reconstruct an input feature sequence of an i-th feature reconstruction network based on a reference feature sequence input into the i-th feature reconstruction network by each of M attention networks in the i-th feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i = 2 … N, N and M are positive integers, and the reference feature sequence conforms to a normal image feature pattern;

[0012] a second feature reconstruction module configured to, if M is equal to 1, perform normalization processing on the first reconstructed feature sequence by a summation and normalization layer to obtain a second reconstructed feature sequence; or, if M is greater than 1, perform weighted summation on M first reconstructed feature sequences by the summation and normalization layer, and perform normalization processing on a result of the weighted summation to obtain the second reconstructed feature sequence;

[0013] a third feature reconstruction module configured to perform processing on the second reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a third reconstructed feature sequence;

[0014] a first determination module configured to determine a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th feature reconstruction network;

[0015] a second determination module configured to determine an abnormal pixel point in the first image based on the feature difference score map of the first image.

[0016] In some implementable manners, the M attention networks comprise at least one of the following: a first attention network and a second attention network; correspondingly, the first feature reconstruction module is configured to perform at least one of the following:

[0017] convert each feature in the input feature sequence in the i-th feature reconstruction network into a first query vector and a first value vector, and convert each feature in the reference feature sequence in the i-th feature reconstruction network into a first key vector by the first attention network; calculate a first attention matrix corresponding to each feature in the input feature sequence in the i-th feature reconstruction network based on the first query vector and the first key vector; and calculate the first reconstructed feature sequence based on the first attention matrix and the first value vector;

[0018] The second attention network is used for converting each feature in the input feature sequence in the i-th feature reconstruction network into a second query vector and converting each feature in the reference feature sequence in the i-th feature reconstruction network into a second key vector and a second value vector; based on the second query vector and the second key vector, a second attention matrix corresponding to each feature in the input feature sequence in the i-th feature reconstruction network is calculated; and based on the second attention matrix and the second value vector, the first reconstructed feature sequence is calculated.

[0019] In some implementations, the first feature reconstruction module is specifically configured to: perform scaled dot product between the first query vector corresponding to the first feature and the transpose of each first key vector to obtain a third attention matrix corresponding to the first feature; and calculate a sum of the third attention matrix corresponding to the first feature and a first mask matrix corresponding to the first feature, and perform normalization processing to obtain the first attention matrix corresponding to the first feature; wherein the values of the first feature corresponding position and the surrounding positions in the first mask matrix are negative infinity, and the values of the remaining positions are 0.

[0020] In some implementations, the first feature reconstruction module is specifically configured to: perform scaled dot product between the second query vector corresponding to the second feature and the transpose of each second key vector to obtain a fourth attention matrix corresponding to the second feature; wherein the second feature is each feature in the input feature sequence in the i-th feature reconstruction network; calculate a sum of the fourth attention matrix corresponding to the second feature and a second mask matrix corresponding to the second feature, and perform normalization processing to obtain a second attention matrix corresponding to the second feature; wherein the values of the second feature corresponding position and the surrounding positions in the second mask matrix are 0, and the values of the remaining positions are negative infinity.

[0021] In some implementations, when the first reconstructed feature sequence output by the first attention network and the first reconstructed feature sequence output by the second attention network are weighted and summed by the sum and normalization layer, the weight corresponding to the first reconstructed feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstructed feature sequence output by the second attention network is a hyperparameter greater than 1.

[0022] In some implementations, the multi-scale feature processing module is specifically configured to: perform multi-scale feature extraction on the first image by a pre-trained model in the feature extraction network to obtain a first feature map under multiple scales; perform fusion on the first feature map under multiple scales by a fusion layer in the feature extraction network to obtain a first fused feature map; and perform shape conversion on the first fused feature map by a conversion layer in the feature extraction network to obtain a first multi-scale feature sequence.

[0023] In some implementable manners, the multi-scale feature processing module is specifically configured to: perform average pooling operations on the first feature maps at the multiple scales respectively through the fusion layer to obtain multiple second feature maps at the same scale; and perform merging in the channel dimension on the multiple second feature maps at the same scale through the fusion layer to obtain the first fusion feature map.

[0024] In some implementable manners, the first determining module is specifically configured to: restore the third reconstruction feature sequence output by the Nth layer feature reconstruction network into third feature maps at multiple scales; restore the first multi-scale feature sequence into fourth feature maps at multiple scales; and perform feature difference score calculation on the third feature maps at multiple scales and the fourth feature maps at multiple scales to obtain a feature difference score map of the first image.

[0025] In some implementable manners, the second determining module is specifically configured to: determine, as an abnormal pixel point in the first image, a pixel point in the feature difference score map whose score is higher than a preset score.

[0026] In a fourth aspect, an embodiment of the present application provides a model training apparatus, comprising: a multi-scale feature processing module, configured to perform multi-scale feature processing on a second image through a feature extraction network to obtain a second multi-scale feature sequence;

[0027] a first feature reconstruction module, configured to reconstruct, through each attention network in M attention networks in an ith layer feature reconstruction network, an input feature sequence of the ith layer feature reconstruction network based on a reference feature sequence input into the ith layer feature reconstruction network to obtain a fourth reconstruction feature sequence, wherein the input feature sequence of the ith layer feature reconstruction network is an output feature sequence of an (i-1)th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the second multi-scale feature sequence, i=2…N, and N and M are positive integers;

[0028] a second feature reconstruction module, configured to, if M is equal to 1, perform normalization processing on the fourth reconstruction feature sequence through a summing and normalization layer to obtain a fifth reconstruction feature sequence; or, if M is greater than 1, perform weighted summation on the M fourth reconstruction feature sequences through the summing and normalization layer, and perform normalization processing on a weighted summation result to obtain the fifth reconstruction feature sequence;

[0029] a third feature reconstruction module, configured to perform processing on the fifth reconstruction feature sequence through other network layers in the ith layer feature reconstruction network to obtain a sixth reconstruction feature sequence;

[0030] a loss calculation module, configured to calculate a loss of a target model constituted by the feature extraction network and the N layer feature reconstruction network based on the second multi-scale feature sequence and the sixth reconstruction feature sequence output by the Nth layer feature reconstruction network.

[0031] The training module is configured to train the target model and the reference feature sequence of each layer in the N-layer feature reconstruction network based on a loss of the target model.

[0032] In some implementations, the multi-scale feature processing module is specifically configured to: perform multi-scale feature extraction on the second image through a pre-trained model in the feature extraction network to obtain a fifth feature map under multiple scales; perform fusion on the fifth feature map under multiple scales through a fusion layer in the feature extraction network to obtain a second fusion feature map; perform shape conversion on the second fusion feature map through a conversion layer in the feature extraction network to obtain a third multi-scale feature sequence, and obtain a second multi-scale feature sequence based on the third multi-scale feature sequence.

[0033] In some implementations, the multi-scale feature processing module is specifically configured to: perform an average pooling operation on the fifth feature map under multiple scales through the fusion layer respectively to obtain multiple sixth feature maps under the same scale; and perform merging in the channel dimension on the multiple sixth feature maps under the same scale through the fusion layer to obtain the second fusion feature map.

[0034] In some implementations, the multi-scale feature processing module is specifically configured to: perform a perturbation operation on the third multi-scale feature sequence to obtain the second multi-scale feature sequence.

[0035] In some implementations, the loss calculation module is specifically configured to: restore the sixth reconstructed feature sequence output by the N-layer feature reconstruction network to a seventh feature map under multiple scales; restore the second multi-scale feature sequence to an eighth feature map under multiple scales; and calculate the loss of the target model based on the seventh feature map under multiple scales and the eighth feature map under multiple scales.

[0036] In some implementations, the loss calculation module is specifically configured to: calculate the loss of the target model under each scale based on the seventh feature map and the eighth feature map under each scale; and sum the loss of the target model under each scale to obtain the total loss of the target model.

[0037] In a fifth aspect, an electronic device is provided, including a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the method in the first aspect to the second aspect or the implementation manners thereof.

[0038] In a sixth aspect, a computer readable storage medium is provided, configured to store a computer program, and the computer program causes a computer to execute the method in the first aspect to the second aspect or the implementation manners thereof.

[0039] In a seventh aspect, a computer program product is provided, including computer program instructions, and the computer program instructions cause a computer to execute the method in the first aspect to the second aspect or the implementation manners thereof.

[0040] In an eighth aspect, a computer program is provided, which causes a computer to execute the method in the first aspect to the second aspect or implementation manners thereof.

[0041] In the technical scheme provided in the present application, since the residual connection is not included on the attention network of the feature extraction network, the possibility of learning shortcut can be reduced. Meanwhile, the present application proposes that the reference feature sequence conforming to the normal image feature mode can be introduced in each layer of the feature reconstruction network, so that the similarity between the abnormal features in the input feature sequence and the reference feature sequence is smaller, and the similarity between the normal features in the input feature sequence and the reference feature sequence is larger. Therefore, the first attention matrix corresponding to each feature in the input feature sequence can reflect the difference between it and different reference features, and the possibility of rank collapse of the attention matrix can be reduced, which is beneficial to improve the reliability of the anomaly detection. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] FIG. 1 is a schematic diagram of a feature reconstruction network of a Transfomer structure;

[0044] FIG. 2 is a schematic diagram of a system architecture related to an embodiment of the present application;

[0045] FIG. 3 is a flowchart of an anomaly detection method provided by an embodiment of the present application;

[0046] FIG. 4 is a schematic diagram of a target model provided by an embodiment of the present application;

[0047] FIG. 5 is a schematic diagram of an average pooling;

[0048] FIG. 6 is a schematic diagram of a first fused feature map;

[0049] FIG. 7 is a schematic diagram of a feature extraction network provided by an embodiment of the present application;

[0050] FIG. 8 is a schematic diagram of a first attention network provided by an embodiment of the present application;

[0051] FIG. 9 is a schematic diagram of a feature reconstruction provided by an embodiment of the present application;

[0052] FIG. 10 is a schematic diagram of a second attention network provided by an embodiment of the present application;

[0053] FIG. 11 is a schematic diagram of another feature reconstruction provided by an embodiment of the present application;

[0054] FIG. 12 is a schematic diagram of another network layer provided by an embodiment of the present application;

[0055] FIG. 13 is a schematic diagram of calculation of a feature difference score map provided by an embodiment of the present application;

[0056] FIG. 14 is a schematic diagram of another target model provided by an embodiment of the present application;

[0057] FIG. 15 is a flowchart of a model training method provided by an embodiment of the present application;

[0058] FIG. 16 is a schematic diagram of another feature extraction network provided by an embodiment of the present application;

[0059] FIG. 17 is a schematic diagram of calculation of a target model loss provided by an embodiment of the present application;

[0060] FIG. 18 is a schematic diagram of another target model provided by an embodiment of the present application;

[0061] FIG. 19 is a schematic diagram of an anomaly detection apparatus 1900 provided by an embodiment of the present application;

[0062] FIG. 20 is a schematic diagram of a model training apparatus 2000 provided by an embodiment of the present application;

[0063] FIG. 21 is a schematic block diagram of an electronic device 2100 provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0065] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0066] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0067] As described above, the current feature reconstruction network is a Transfomer structure, which has a learning shortcut problem, in other words, the structure is not easy to learn the normal image feature mode, so that for the actual abnormal image, the feature reconstruction network through the structure is not easy to reconstruct the feature sequence conforming to the normal image feature mode, and further leads to the problem that the anomaly detection based on the difference between the multi-scale feature sequence and the reconstructed feature sequence has low reliability.

[0068] FIG. 1 is a schematic diagram of a feature reconstruction network of a Transfomer structure, as shown in FIG. 1, the feature reconstruction network is composed of N layers of encoders and N layers of decoders. Each layer of the encoder includes an attention network, a feed forward network (FFN), and an ADD&Normalize layer. Each layer of the decoder also includes an attention network, a feed forward network, and an ADD&Normalize layer.

[0069] The attention network in each layer of the encoder can calculate, according to three preset weight matrices WQ, WK and WV, a Query (Q) vector, a Key (K) vector and a Value (V) vector corresponding to each feature in the input feature sequence of the layer, calculate an attention matrix corresponding to each feature based on the Q vector and the K vector, wherein the position information can be combined when calculating the attention matrix, and then calculate the product of the attention matrix and the V vector to obtain a product result. The residual and normalization layer of the encoder layer can sum the product result and the input feature sequence of the layer to obtain a sum result, and then normalize the sum result. The feedforward network of the encoder layer linearly and nonlinearly transforms the normalized result, and then passes through a residual and normalization layer to obtain an encoding sequence corresponding to the input feature sequence of the layer. Finally, the encoding sequence output by the Nth layer of the encoder can be used as the input feature sequence of the Nth layer of the decoder.

[0070] The attention network, the residual and the normalization layer in each layer of the decoder can process the multi-scale feature sequence to obtain a normalized result, wherein the position information can be combined when calculating the attention matrix. Further, the input feature sequence of the layer and the normalized result can be merged, and the merged result is processed again by the attention network, the residual and the normalization layer of the decoder layer to obtain a normalized result again. Further, the feedforward network of the decoder layer can linearly and nonlinearly transform the normalized result, and then pass through a residual and normalization layer to obtain a decoding sequence corresponding to the input feature sequence of the layer. Finally, the decoding sequence output by the Nth layer of the decoder can be used as the reconstructed feature sequence corresponding to the multi-layer feature sequence.

[0071] It should be understood that the learning shortcut problem caused by the current Transfomer structure is mainly caused by the residual connection in the structure, as shown in FIG. 1, especially the residual connection above the attention network, because the residual connection makes the output of the attention network include the input of the attention network, forming a learning shortcut, making it difficult for the attention network to learn the working mode from the input to the output, and in the anomaly detection scenario, making it difficult for the attention network to learn the normal image feature mode, that is, making it difficult for the Transfomer structure to learn the normal image feature mode, so that for the actual abnormal image, it is difficult for the feature reconstruction network of the structure to reconstruct a feature sequence conforming to the normal image feature mode, and further causing the anomaly detection based on the difference between the multi-scale feature sequence and the reconstructed feature sequence to have low reliability.

[0072] In order to solve the above technical problems, the application proposes not including a residual connection on the attention network of the feature extraction network. Considering that not setting the residual connection may cause the rank collapse of the attention matrix, the application also proposes that a reference feature sequence can be introduced at each layer. Since the reference feature sequence conforms to the normal image feature mode, the similarity between the abnormal features in the input feature sequence and the reference feature sequence is small, and the similarity between the normal features in the input feature sequence and the reference feature sequence is large. Therefore, the attention matrix corresponding to each feature in the input feature sequence can reflect the difference between it and different reference features, thereby reducing the possibility of rank collapse of the attention matrix, and improving the reliability of anomaly detection.

[0073] In some embodiments, the system architecture of the embodiments of the application is shown in FIG. 2.

[0074] FIG. 2 is a schematic diagram of a system architecture related to an embodiment of the application, which involves a user device 201, a data collection device 202, a training device 203, an execution device 204, a database 205 and a content library 206.

[0075] The data collection device 202 is configured to read training data from the content library 206 and store the read training data in the database 205. The training data related to the embodiments of the application includes normal images.

[0076] The training device 203 trains the target model based on the training data maintained in the database 205, so that the trained target model can effectively perform feature reconstruction. The target model includes a feature extraction network and an N-layer feature reconstruction network.

[0077] In addition, referring to FIG. 2, the execution device 204 is configured with an I / O interface 207 to interact with external devices. For example, the I / O interface receives an image sent by the user device 201. The computing module 208 in the execution device 204 processes the image input by the user based on the trained target model to obtain a reconstructed feature sequence. Illustratively, the feature extraction module in the target model performs feature processing on the input image, and outputs a multi-scale feature sequence corresponding to the image. Then, the N-layer feature reconstruction network in the target model can reconstruct the multi-scale feature sequence to obtain a reconstructed feature sequence. Finally, through the anomaly detection module 209 in the computing module 208, the feature difference score map of the image is determined based on the reconstructed feature sequence and the multi-scale feature sequence; based on the feature difference score map of the image, the abnormal pixel points in the image are determined. Finally, the abnormal detection result, such as the coordinates of the abnormal pixel points, is sent to the user device 201 through the I / O interface 207.

[0078] The user device 201 can include a mobile phone, a tablet computer, a notebook computer, a palm computer, a mobile internet device (MID), a desktop computer, or other terminal device with a browser function.

[0079] The execution device 204 can be a server.

[0080] For example, the server can be a rack server, a blade server, a tower server, or a cabinet server, or other computing device. The server can be a standalone test server, or a test server cluster composed of multiple test servers.

[0081] In this embodiment, the execution device 204 is connected with the user device 201 through a network. The network can be an intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4rd Generation (4G) network, a 5rd Generation (5G) network, Bluetooth, wireless fidelity (Wi-Fi), a voice network, or other wireless or wired network.

[0082] It should be noted that FIG. 2 is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, elements, modules, etc. shown in the diagram does not constitute any limitation. In some embodiments, the data collection device 202, the user device 201, the training device 203, and the execution device 204 can be the same device. The database 205 can be distributed on one server or multiple servers, and the content library 206 can be distributed on one server or multiple servers.

[0083] The embodiments of the present application will be described in detail below:

[0084] FIG. 3 is a flowchart of an anomaly detection method provided by the embodiments of the present application. The method can be executed by an electronic device, which can be the execution device 204 in FIG. 2, or the electronic device 2100 shown in FIG. 21, but is not limited thereto. In the embodiments of the present application, the execution subject is taken as an example for description. As shown in FIG. 3, the method can include the following steps.

[0085] S310: performing multi-scale feature processing on the first image by the feature extraction network to obtain a first multi-scale feature sequence;

[0086] S320: reconstructing, by each of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on a reference feature sequence input into the i-th feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i=2…N, N and M are positive integers, and the reference feature sequence conforms to a normal image feature mode;

[0087] S330: if M is equal to 1, performing normalization processing on the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, performing weighted sum on the M first reconstructed feature sequences by the sum and normalization layer, and performing normalization processing on a weighted sum result to obtain the second reconstructed feature sequence;

[0088] S340: processing the second reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a third reconstructed feature sequence;

[0089] S350: determining a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th feature reconstruction network;

[0090] S360: determining an abnormal pixel point in the first image based on the feature difference score map of the first image.

[0091] In the embodiments of the present application, the model composed of the feature extraction network and the N-layer feature reconstruction network can be referred to as a target model, and the structure of the target model will be described as follows:

[0092] FIG. 4 is a schematic diagram of a target model provided by an embodiment of the present application. As shown in FIG. 4, the feature extraction network can perform multi-scale feature processing on the image to obtain a first multi-scale feature sequence. Each feature extraction network in the N-layer feature reconstruction network includes an attention network, a sum and normalization layer, and other network layers. In FIG. 4, for example, each layer of the feature reconstruction network includes two attention networks. Each attention network can reconstruct the input feature sequence of the layer of the feature reconstruction network based on the reference feature sequence in the layer of the feature reconstruction network to obtain a first reconstructed feature sequence. If M is equal to 1, the sum and normalization layer in each layer of the feature reconstruction network normalizes the first reconstructed feature sequence to obtain a second reconstructed feature sequence. If M is greater than 1, the sum and normalization layer in each layer of the feature reconstruction network performs weighted summation on the M first reconstructed feature sequences and normalizes the weighted summation result to obtain a second reconstructed feature sequence. The other network layers in each layer of the feature reconstruction network process the second reconstructed feature sequence to obtain a third reconstructed feature sequence.

[0093] The S310 is described in detail as follows:

[0094] In some implementable manners, the execution device can capture the to-be-detected article by an image acquisition device to obtain the first image. The image acquisition device can be an image acquisition unit in the execution device, such as a camera, or can be a peripheral device of the execution device, and the present application does not limit the same.

[0095] It should be understood that the S310 can be implemented by any one of the following implementable manners, but is not limited thereto:

[0096] Implementable manner one, the S310 can include:

[0097] S310-1A: performing multi-scale feature extraction on the first image by a pre-trained model in the feature extraction network to obtain a first feature map under multi-scale;

[0098] In some implementable manners, the pre-trained model can be a convolutional neural network (CNN) including a plurality of convolutional layers, for example, a ResNet-50 model, a ResNet-34 model, an EfficientNet, a VGG-160 model, and the like. Each convolutional layer in the CNN can extract key feature information of the first image, wherein the key feature information is output in the form of a feature map, and each element in the feature map is referred to as a feature point. In the present application, the feature map extracted by the first image through each convolutional layer is referred to as a first feature map.

[0099] It should be understood that each first feature map is the result of processing the first image by a specific convolution kernel, i.e., a filter, which can capture certain features or patterns of the first image. Among them, different convolution kernels can capture different types of features, such as edges, corner points, textures, etc. With the increase of network layers, the "depth" of the feature map will also increase. The feature map of the shallower layer tends to capture more basic and general features, such as edges and simple textures, while the feature map of the deeper layer can represent more complex and high-level features, such as specific parts of an object or complex shapes.

[0100] It should be understood that assuming the first image is a two-dimensional image, the feature map obtained after processing by the convolution kernel is still a two-dimensional structure, but in the convolution network, in order to capture more features, each convolution layer usually uses multiple convolution kernels, each of which generates a two-dimensional feature map. These feature maps are stacked together to form a three-dimensional feature body, where the third dimension (depth) corresponds to the number of different convolution kernels. Therefore, the first feature map output by each convolution layer can be a three-dimensional structure, where assuming the first feature map output by the i-th convolution layer is denoted as f i , then H i and W i respectively represent the height and width of the first feature map f i , and D i represents the number of convolution kernels or the number of channels of the i-th convolution layer.

[0101] S310-2A: fusing the first feature maps under multiple scales through a fusion layer in the feature extraction network to obtain a first fused feature map;

[0102] It should be understood that by fusing the first feature maps under multiple scales, the image features can be enhanced, and thus the robustness of the anomaly detection can be improved.

[0103] In some implementable manners, the execution device can perform an average pooling operation on the first feature maps under multiple scales through the fusion layer to obtain multiple second feature maps under the same scale; and perform a merging in the channel dimension on the multiple second feature maps under the same scale through the fusion layer to obtain the first fused feature map.

[0104] It should be understood that in average pooling, the average value of all values in the pooling window is used as the output of the window. Average pooling helps to retain the background information of the image because it considers all pixel values within the window.

[0105] wherein the average pooling comprises the following steps:

[0106] 1. Define a pooling window:

[0107] Select a fixed-size window (e.g., 2*2, 3*3, etc.) that will slide over the feature map.

[0108] 2. Define the stride:

[0109] Define the stride at which the pooling window slides over the feature map. The stride is usually the same as the window size, but it can also be different.

[0110] 3. Padding:

[0111] In some cases, to maintain the size of the feature map, additional zero values can be added to the edges of the feature map (known as zero padding).

[0112] 4. Perform pooling:

[0113] For each position on the feature map, place the pooling window at that position and calculate the values within the window according to the average pooling type, then place the result in the corresponding position of the output feature map.

[0114] For example, FIG. 5 is a schematic diagram of an average pooling, as shown in FIG. 5, a 2*2 pooling window is selected for the left feature map in FIG. 5, the window is slid with a 2*2 stride, the average value of all values under each window is calculated, and finally the right feature map is output. The left feature map in FIG. 5 is the first feature map, and the right feature map is the second feature map.

[0115] Suppose the first feature map f i is processed by average pooling to obtain the second feature map g i , and the first fused feature map is denoted as G, then G∈R H*W*D , where H and W represent the height and width of the second feature map g i , respectively, and D i represents the number of convolutional kernels or channels of the i-th convolutional layer, and D represents the sum of the number of convolutional kernels or channels of all second feature maps. For example, FIG. 6 is a schematic diagram of a first fused feature map, as shown in FIG. 6, the length, width, and height of the first fused feature map are H, W, and D, respectively.

[0116] In some implementations, the execution device can perform padding or cropping operations on the first feature maps under multiple scales through the fusion layer to obtain multiple second feature maps under the same scale, and perform merging in the channel dimension on the multiple second feature maps under the same scale through the fusion layer to obtain the first fused feature map.

[0117] For example, assuming that the size of the second feature map is specified as 4*4, for the first feature map with a size smaller than 4*4, feature points with a value of 0 can be filled at the edges thereof to change the size thereof to 4*4, for the first feature map with a size larger than 4*4, edge feature points thereof can be cropped to change the size thereof to 4*4, and for the first feature map with a size equal to 4*4, no operation can be performed thereon.

[0118] It should be understood that embodiments of the present application do not limit the calculation manner of the first fused feature map.

[0119] S310-3A: performing shape conversion on the first fused feature map through a conversion layer in the feature extraction network to obtain a first multi-scale feature sequence.

[0120] In some implementable manners, assuming that the first fused feature map is a three-dimensional structure with a size of H*W*D, the execution device can convert the first fused feature map into H*W D-dimensional vectors, and sort the H*W D-dimensional vectors in a preset order to obtain the first multi-scale feature sequence.

[0121] In some implementable manners, the preset order is an arrangement order from left to right and then from top to bottom, but is not limited thereto.

[0122] FIG. 7 is a schematic diagram of a feature extraction network provided by an embodiment of the present application, as shown in FIG. 7, after the first feature map under multi-scale is processed, a first multi-scale feature sequence is formed, wherein each box in the first multi-scale feature sequence represents a D-dimensional vector.

[0123] Implementable manner two, S310 can include:

[0124] S310-1B: inputting the first image and a size requirement to be met by the first multi-scale feature sequence into a neural network model to obtain the first multi-scale feature sequence.

[0125] In some implementable manners, the neural network model can be trained through training samples, each training sample can include: an image, a size requirement to be met by a multi-scale feature sequence corresponding to the image, and an actual multi-scale feature sequence, wherein the actual multi-scale feature sequence can be used as a sample label. The training device can use a supervised training manner, for example, it can input the image and the size requirement to be met by the multi-scale feature sequence corresponding to the image into the neural network model to output a predicted multi-scale feature sequence. Further, the training device can calculate a loss based on the actual multi-scale feature sequence and the predicted multi-scale feature sequence included in all training samples, adjust parameters of the neural network model based on the loss, and stop training when the number of training reaches a preset number or the loss reaches a minimum value.

[0126] In some implementable manners, when training the neural network model, the loss function utilized by the training device can be any one of, but not limited to, an L1 loss function, a Mean Squared Error (MSE) loss function, a Cross Entropy Loss function, and the like.

[0127] Based on this, the execution device can input the first image and the size requirement that the first multi-scale feature sequence needs to meet into the neural network model to obtain the first multi-scale feature sequence.

[0128] The following describes S320 in detail:

[0129] In some implementable manners, the M attention networks include at least one of the following: a first attention network and a second attention network. Correspondingly, by each of the M attention networks in the i-th layer feature reconstruction network, the input feature sequence of the i-th layer feature reconstruction network is reconstructed based on the reference feature sequence input into the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, including at least one of the following:

[0130] 1. By the first attention network, each feature in the input feature sequence in the i-th layer feature reconstruction network is converted into a first query vector and a first value vector, and each feature in the reference feature sequence in the i-th layer feature reconstruction network is converted into a first key vector; based on the first query vector and the first key vector, a first attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated; and based on the first attention matrix and the first value vector, the first reconstructed feature sequence is calculated.

[0131] 2. By the second attention network, each feature in the input feature sequence in the i-th layer feature reconstruction network is converted into a second query vector, and each feature in the reference feature sequence in the i-th layer feature reconstruction network is converted into a second key vector and a second value vector; based on the second query vector and the second key vector, a second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated; and based on the second attention matrix and the second value vector, the first reconstructed feature sequence is calculated.

[0132] The following describes the case of obtaining the first reconstructed feature sequence by the first attention network:

[0133] For example, FIG. 8 is a schematic diagram of a first attention network provided by an embodiment of the present application. As shown in FIG. 8, the execution device can convert each feature in the input feature sequence in the i-th layer feature reconstruction network into a first query vector and a first value vector, and convert each feature in the reference feature sequence in the i-th layer feature reconstruction network into a first key vector through the first attention network. For each feature in the input feature sequence, the execution device can calculate a first attention matrix corresponding to the feature based on the first query vector corresponding to the feature and the first key vector corresponding to each feature in the input feature sequence, in other words, each feature in the input feature sequence corresponds to a first attention matrix. Further, the execution device can calculate the product of the first attention matrix and the matrix composed of all first value vectors to obtain the reconstruction feature corresponding to the feature, and the reconstruction features corresponding to all features in the input feature sequence form a first reconstruction feature sequence.

[0134] It should be understood that, as shown in FIG. 8, the input feature sequence in the first layer feature reconstruction network is a first multi-scale feature sequence, wherein each box represents a feature, and the dimension of the reference feature sequence of each layer is the same as the dimension of the input feature sequence of the layer.

[0135] It should be understood that, for each feature in the input feature sequence, the execution device calculates the product of the first attention matrix corresponding to the feature and the matrix composed of all first value vectors through the first attention network as follows: the execution device multiplies each attention value in the first attention matrix with the first value vector at the corresponding position in the matrix composed of all first value vectors through the first attention network to obtain the multiplication result at each position, and then adds the multiplication results to obtain the reconstruction feature corresponding to the feature.

[0136] It should be understood that, since the first key vector is converted from the feature in the reference feature sequence, and the reference feature sequence conforms to the normal image feature mode, when the first attention matrix is calculated based on the first key vector and the first query vector, the similarity between the feature corresponding to the abnormal region in the input feature sequence and the first key vector is small, therefore, the first reconstruction feature sequence usually does not retain the abnormal image feature mode, and the normal image feature mode is retained with high probability, so that the difference between the input feature sequence and the reconstruction feature sequence is large, which is further conducive to anomaly detection.

[0137] In some implementable manners, for each feature in the input feature sequence, the execution device can calculate the first query vector corresponding to the feature by multiplying the query weight matrix WQ and the feature, which is a D-dimensional vector, through the first attention network.

[0138] In some implementable manners, for each feature in the input feature sequence, the execution device can calculate the value weight matrix W V and the feature, i.e., a D-dimensional vector, to obtain the first value vector corresponding to the feature through the first attention network.

[0139] In some implementable manners, for each feature in the reference feature sequence, the execution device can calculate the key weight matrix W K and the feature, i.e., a D-dimensional vector, to obtain the first key vector corresponding to the feature through the first attention network.

[0140] Suppose each feature in the input feature sequence in the i-th layer feature reconstruction network is called a first feature, the execution device can obtain the first attention matrix corresponding to the first feature through the first attention network in any one of the following implementable manners, but is not limited thereto:

[0141] In some implementable manners, the execution device can obtain the first attention matrix corresponding to the first feature by performing scaled dot product between the first query vector corresponding to the first feature and the transpose of each first key vector, and performing normalization processing through the first attention network.

[0142] Suppose the first query vector corresponding to the first feature is denoted as Q 1, the first key vector corresponding to each feature is denoted as K 1, and the transpose of the first key vectors K 1 is denoted as K, then the first attention matrix is where d K1 represents the dimension of K 1, where Q 1 · K is a matrix, and each element in the matrix represents the dot product result of Q 1 and the transpose of a K 1.

[0143] In some implementable manners, the execution device can obtain the first attention matrix corresponding to the first feature by performing scaled dot product between the first query vector corresponding to the first feature and the transpose of each first key vector through the first attention network, calculating the sum of the third attention matrix corresponding to the first feature and the first mask matrix corresponding to the first feature, and performing normalization processing; wherein the values of the first feature corresponding position and the surrounding positions in the first mask matrix are negative infinity, and the values of the remaining positions are 0.

[0144] wherein the first mask matrix can also be called a neighbor masked attention matrix or an attention map, which is used to mask the attention values corresponding to the first feature and its neighbor features in the third attention matrix corresponding to the first feature.

[0145] In some implementable manners, the neighbor features of the first feature can include at least one of, but are not limited to, a feature directly above the first feature, a feature directly below the first feature, a feature to the left of the first feature, a feature to the right of the first feature, a feature at the top-left corner of the first feature, a feature at the bottom-left corner of the first feature, a feature at the top-right corner of the first feature, and a feature at the bottom-right corner of the first feature.

[0146] It should be understood that the surrounding positions of the corresponding position of the first feature refer to the corresponding positions of the neighbor features of the first feature in the first mask matrix.

[0147] Suppose the first query vector is denoted as Q1, the first key vector is denoted as K1, a matrix composed of the transposes of the first key vectors K1 is denoted as K, and the first attention matrix is denoted as M1, then the first attention matrix is where d K1 represents the dimension of K1.

[0148] For example, FIG. 9 is a schematic diagram of feature reconstruction provided by an embodiment of the present application. As shown in FIG. 9, for the first attention network in the i-th layer feature reconstruction network, the input feature sequence and the reference feature sequence can be represented by matrixes, respectively, where each box in the matrix can represent a feature, which can be a D-dimensional vector. The execution device can convert the first feature into Q1 through the first attention network, and convert each feature in the input feature sequence into V1 and each feature in the reference feature sequence into K1. Further, the execution device can perform scaled dot product on Q1 and the transpose of each K1 through the first attention network to obtain a third attention matrix A1. The execution device can calculate the sum of the third attention matrix A1 and a first mask matrix M1, and perform normalization processing through a Softmax function to obtain a first attention matrix A2. As shown in FIG. 9, the values of the corresponding positions of the first feature and the surrounding positions in M1 are negative infinity, and the values of the other positions are 0. The values of the corresponding positions of the feature directly above the first feature, the feature directly below the first feature, the feature to the left of the first feature, the feature to the right of the first feature, the feature at the top-left corner of the first feature, the feature at the bottom-left corner of the first feature, the feature at the top-right corner of the first feature, and the feature at the bottom-right corner of the first feature are negative infinity, and the values of the other positions are 0. Further, the execution device can calculate the product of the first attention matrix A2 and a matrix composed of all V1 through the first attention network to obtain the reconstructed feature corresponding to the first feature.

[0149] It should be understood that the shortcut learning problem is mainly caused by the fact that the output of the network directly includes the input of the network. Based on this, the embodiments of the present application propose that the attention values corresponding to the first feature and its neighbor features in the third attention matrix corresponding to the first feature can be masked, so that when the first feature is reconstructed, the first value vector corresponding to the first feature and its neighbor features can be ignored, thereby preventing the reconstructed feature from including the first feature and its neighbor features, reducing the possibility of shortcut learning, and further improving the robustness of the anomaly detection method.

[0150] The following describes the case where the first reconstructed feature sequence is obtained through the second attention network:

[0151] For example, FIG. 10 is a schematic diagram of a second attention network provided by the embodiments of the present application. As shown in FIG. 10, the execution device can convert each feature in the input feature sequence in the i-th layer feature reconstruction network into a second query vector and each feature in the reference feature sequence in the i-th layer feature reconstruction network into a second key vector and a second value vector through the second attention network; for each feature in the input feature sequence, the second attention matrix corresponding to the feature can be calculated based on the second query vector corresponding to the feature and the second key vector corresponding to each feature in the reference feature sequence, in other words, each feature in the input feature sequence corresponds to a second attention matrix; further, the product of the second attention matrix and the matrix composed of all second value vectors can be calculated to obtain the reconstructed feature corresponding to the feature, and the reconstructed features corresponding to all features in the input feature sequence form the first reconstructed feature sequence.

[0152] It should be understood that, as shown in FIG. 10, the input feature sequence in the first layer feature reconstruction network is the first multi-scale feature sequence, wherein each box represents a feature, and the dimension of the reference feature sequence of each layer is the same as the dimension of the input feature sequence of the layer.

[0153] It should be understood that for each feature in the input feature sequence, the execution device performs the multiplication operation of the second attention matrix corresponding to the feature and the matrix composed of all second value vectors through the second attention network as follows: the execution device multiplies each attention value in the second attention matrix and the second value vector at the corresponding position in the matrix composed of all second value vectors through the second attention network to obtain the multiplication result at each position, and then adds these multiplication results to obtain the reconstructed feature corresponding to the feature.

[0154] It should be understood that, since the second key vector is converted from the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature mode, so that when the second attention matrix is calculated based on the second key vector and the second query vector, the abnormal features in the input feature sequence, i.e., the features corresponding to the abnormal region, have a smaller similarity with the second key vector, therefore, the first reconstructed feature sequence usually does not retain the abnormal image feature mode, and the normal image feature mode is more likely to be retained, so that the difference between the input feature sequence and the reconstructed feature sequence is larger, and further, it is beneficial to the anomaly detection. In addition, since the second value vector is also converted from the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature mode, so that when the feature reconstruction is performed based on the second value vector, the first reconstructed feature sequence generated is more likely to retain the normal image feature mode, so that the difference between the input feature sequence and the reconstructed feature sequence is larger, and further, the robustness of the anomaly detection method is further improved.

[0155] In some implementable manners, for each feature in the input feature sequence, the execution device can calculate the second query vector corresponding to the feature by multiplying the query weight matrix WQ and the feature, which is a D-dimensional vector, through the second attention network.

[0156] In some implementable manners, for each feature in the reference feature sequence, the execution device can calculate the second value vector corresponding to the feature by multiplying the value weight matrix WV and the feature, which is a D-dimensional vector, through the second attention network.

[0157] In some implementable manners, for each feature in the reference feature sequence, the execution device can calculate the second key vector corresponding to the feature by multiplying the key weight matrix WK and the feature, which is a D-dimensional vector, through the second attention network.

[0158] Suppose each feature in the input feature sequence in the i-th layer feature reconstruction network is called a second feature, then the execution device obtains the second attention matrix corresponding to the second feature through the second attention network in any one of the following implementable manners, but is not limited thereto:

[0159] In some implementable manners, the execution device performs scaled dot product between the second query vector corresponding to the second feature and the transpose of each second key vector through the second attention network, and performs normalization processing to obtain the second attention matrix corresponding to the second feature.

[0160] Suppose the second query vector corresponding to the second feature is denoted as Q2, and the second key vector corresponding to each feature is denoted as K2,

[0161] The matrix composed of the transposes of these second key vectors K2 is denoted as K′, and the second attention matrix is wherein d K2 denotes the dimensionality of K2, wherein Q2·K' is a matrix whose elements represent the dot product results of Q2 and a transpose of K2.

[0162] In some implementable manners, the execution device performs, by the second attention network, scaled dot product of a second query vector corresponding to the second feature and a transpose of each second key vector to obtain a fourth attention matrix corresponding to the second feature; and calculates a sum of the fourth attention matrix corresponding to the second feature and a second mask matrix corresponding to the second feature, and performs normalization processing to obtain a second attention matrix corresponding to the second feature; wherein values of the second feature corresponding position and surrounding positions in the second mask matrix are 0, and values of other positions are negative infinity.

[0163] wherein the second mask matrix is used to mask attention values corresponding to other features except the second feature and neighbor features of the second feature in the fourth attention matrix corresponding to the second feature.

[0164] In some implementable manners, the neighbor features of the second feature can include at least one of, but are not limited to, a feature directly above the second feature, a feature directly below the second feature, a feature to the left of the second feature, a feature to the right of the second feature, a feature at the top left corner of the second feature, a feature at the bottom left corner of the second feature, a feature at the top right corner of the second feature, and a feature at the bottom right corner of the second feature.

[0165] It should be understood that the surrounding positions of the second feature corresponding position refer to the corresponding positions of the neighbor features of the second feature in the second mask matrix.

[0166] Suppose the second query vector is denoted as Q2, the second key vector is denoted as K2, the matrix composed of the transposes of these second key vectors K2 is denoted as K', and the second attention matrix is denoted as M2, then the second attention matrix is wherein d K2 denotes the dimensionality of K2.

[0167] For example, FIG. 11 is a schematic diagram of another feature reconstruction provided by an embodiment of the present application. As shown in FIG. 11, for a second attention network in the i-th layer feature reconstruction network, the input feature sequence and the reference feature sequence can be represented by a matrix, respectively, where each box in the matrix can represent a feature, which can be a D-dimensional vector. The second feature can be converted into Q2 by the second attention network, and each feature in the reference feature sequence can be converted into V2 and K2 by the second attention network. Further, the second attention network can perform a scaled dot product between Q2 and the transpose of each K2 to obtain a fourth attention matrix A3. The sum of the fourth attention matrix A3 and a second mask matrix M2 is calculated, and normalized by a Softmax function to obtain a second attention matrix A4. The values of the second feature and the surrounding positions in M2 are 0, and the values of the other positions are negative infinity. As shown in FIG. 11, the values of the features above, below, left, right, top-left, bottom-left, top-right, and bottom-right of the second feature are 0, and the values of the other positions are negative infinity. Further, the second attention network can calculate the product of the second attention matrix A2 and the matrix composed of all V2 to obtain the reconstructed feature corresponding to the second feature.

[0168] It should be understood that, since the second value vector is the result of converting the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, the present embodiment proposes to mask the attention values of the features corresponding to the fourth attention matrix of the second feature except the second feature and its neighbor features, so that when reconstructing the second feature, the second value vector corresponding to the second feature and its neighbor features can be mainly referenced, thereby further improving the robustness of the anomaly detection method.

[0169] It should be understood that, although the first attention network solves the learning shortcut problem to some extent, it is still possible that abnormal features are left in the first reconstructed feature sequence output by the first attention network, because in the first attention network, the value vector corresponds to the features in the input feature sequence, and the input feature sequence can contain abnormal features. In order to solve this problem, the present embodiment proposes to reconstruct the features by the second attention network, because in the second attention network, the value vector corresponds to the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, so that the possibility of abnormal features existing in the first reconstructed feature sequence output by the second attention network is low, thereby improving the robustness of the anomaly detection method.

[0170] It should be understood that the embodiments of the present application can refer to the attention mechanism adopted by the first attention network in combination with the first mask matrix as Masked Learnable Key Attention (MLKA), and refer to the attention mechanism adopted by the second attention network in combination with the second mask matrix as Local Cross Attention (LCA). The embodiments of the present application can adopt at least one of the two attention mechanisms.

[0171] It should be understood that through the combination of MLKA and LCA, on the one hand, since in the attention network based on the LCA mechanism, the value vector corresponds to the feature in the reference feature sequence, and the reference feature sequence conforms to the normal image feature mode, so that the possibility of the abnormal feature existing in the first reconstructed feature sequence output by the attention network is low, thereby the robustness of the anomaly detection method can be improved; on the other hand, since in the attention network based on the LCA mechanism, the attention values corresponding to the features other than the second feature and its neighbor features in the fourth attention matrix corresponding to the second feature are masked, so that when reconstructing the second feature, the second value vector corresponding to the second feature and its neighbor features can be mainly referred to, thereby the robustness of the anomaly detection method can be further improved.

[0172] It should be understood that in the case of the combination of the first attention network and the second attention network, or in the case of the combination of the first attention network based on the MLKA mechanism and the first attention network based on the LCA mechanism, for each feature in the input feature sequence, the first attention network can reconstruct the feature to obtain a reconstructed feature, similarly, the second attention network can also reconstruct the feature to obtain a reconstructed feature, and the summation and normalization layer can perform weighted summation on the two reconstructed features, and then perform normalization processing to obtain a reconstructed feature corresponding to the feature.

[0173] It should be noted that the first feature can be each feature of the input feature sequence, and the second feature can also be each feature of the input feature sequence. The first feature and the second feature can be the same feature, or can be different features. For example, the first feature and the second feature shown in FIGS. 9 and 10 are not the same feature, but in the case of the combination of the first attention network and the second attention network, or in the case of the combination of the first attention network based on the MLKA mechanism and the first attention network based on the LCA mechanism, the first feature and the second feature refer to the same feature.

[0174] The S330 will be described in detail below:

[0175] In some implementable manners, if M is equal to 1, the execution device normalizes the first reconstructed feature sequence by the summing and normalizing layer to obtain a second reconstructed feature sequence.

[0176] For example, if the N-layer feature reconstruction network only includes the first attention network adopting the MLKA mechanism, the execution device only normalizes the first reconstructed feature sequence output by the first attention network by the summing and normalizing layer to obtain a second reconstructed feature sequence.

[0177] In some implementable manners, if M is greater than 1, the execution device performs weighted summation on the M first reconstructed feature sequences by the summing and normalizing layer to obtain a weighted summation result, and normalizes the weighted summation result to obtain a second reconstructed feature sequence.

[0178] For example, if the N-layer feature reconstruction network includes the first attention network adopting the MLKA mechanism and the second attention network adopting the LCA mechanism, the execution device performs weighted summation on the first reconstructed feature sequences output by the two attention networks by the summing and normalizing layer to obtain a weighted summation result, and normalizes the weighted summation result to obtain a second reconstructed feature sequence.

[0179] In some implementable manners, when performing weighted summation on the first reconstructed feature sequence output by the first attention network and the first reconstructed feature sequence output by the second attention network by the summing and normalizing layer, the weight corresponding to the first reconstructed feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstructed feature sequence output by the second attention network is a hyperparameter greater than 1.

[0180] For example, if the N-layer feature reconstruction network includes the first attention network adopting the MLKA mechanism and the second attention network adopting the LCA mechanism, when performing weighted summation on the first reconstructed feature sequences output by the two attention networks, the weight corresponding to the first attention network adopting the MLKA mechanism can be set to 1, and the weight corresponding to the second attention network adopting the LCA mechanism can be set to a hyperparameter α greater than 1, so as to ensure that the target model pays more attention to the LCA.

[0181] It should be understood that the present embodiment does not limit the size of the weight corresponding to the first attention network adopting the MLKA mechanism and the weight corresponding to the second attention network adopting the LCA mechanism.

[0182] The S340 is described in detail as follows:

[0183] In some implementable manners, the other network layer in the i-th layer feature reconstruction network comprises a feedforward network layer and another summation and normalization layer, based on which the performing device processes the second reconstructed feature sequence through the other network layer in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence, comprising: the performing device performs linear and nonlinear transformation on the second reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain a first transformation result, and then performs summation on the first transformation result and the second reconstructed feature sequence through the summation and normalization layer, and performs normalization processing on the summation result to obtain the third reconstructed feature sequence.

[0184] For example, FIG. 12 is a schematic diagram of the other network layer provided by the embodiment of the present application, as shown in FIG. 12, the performing device obtains the third reconstructed feature sequence after processing through the feedforward network and the summation and normalization layer.

[0185] In some implementable manners, the other network layer in the i-th layer feature reconstruction network comprises a feedforward network layer, based on which the performing device processes the second reconstructed feature sequence through the other network layer in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence, comprising: the performing device performs linear and nonlinear transformation on the second reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain the third reconstructed feature sequence.

[0186] It should be understood that the embodiment of the present application does not limit the other network layer in the i-th layer feature reconstruction network, and does not limit the corresponding processing of the other network layer.

[0187] The S350 is described in detail as follows:

[0188] It should be understood that the embodiment of the present application can implement the S350 through any one of the following implementable manners, but is not limited thereto:

[0189] In some implementable manners, the S350 can comprise:

[0190] S350-1A: restoring the third reconstructed feature sequence output by the N-th layer feature reconstruction network to the third feature map under the multi-scale;

[0191] It should be understood that the third reconstructed feature sequence output by the Nth layer feature reconstruction network is the final reconstructed feature sequence corresponding to the first multi-scale feature sequence, and therefore the third reconstructed feature sequence has the same feature ordering manner as the first multi-scale feature sequence, and the first multi-scale feature sequence is a sequence of H*W D-dimensional vectors, and based on this, the third reconstructed feature sequence is also a sequence of H*W D-dimensional vectors. Based on this, the execution device can convert the third reconstructed feature sequence into a feature map of an H*W*D three-dimensional structure, and then convert the feature map into a feature map corresponding to each convolutional layer in the Nth layer feature reconstruction network according to the number and size of the convolutional kernel used by each convolutional layer, wherein the feature map corresponding to the ith convolutional layer is a three-dimensional structure of H i *W i *D i .

[0192] S350-2A: restoring the first multi-scale feature sequence to a fourth feature map under multi-scale;

[0193] It should be understood that the same restoration method as the third reconstructed feature sequence described above can be used for the first multi-scale feature sequence to obtain the fourth feature map under multi-scale.

[0194] S350-3A: performing feature difference score calculation on the third feature map under multi-scale and the fourth feature map under multi-scale to obtain a feature difference score map of the first image.

[0195] It should be understood that the feature difference score map is also referred to as a feature difference score matrix, and each element in the feature difference score matrix represents the feature difference score of the pixel point corresponding to the element.

[0196] In some implementations, the execution device can perform feature difference score calculation on the third feature map under multi-scale and the fourth feature map under multi-scale using the mean square error (MSE) method to obtain a feature difference score map of the first image.

[0197] For example, assume that the jth third feature map is denoted as the jth fourth feature map is denoted as and the feature difference score map of the first image is denoted as S, then the feature difference score map can be calculated by the following formula (1):

[0198] wherein L in formula (1) represents the number of convolutional layers in the feature extraction network, or the number of feature extraction layers, represents the subtraction of corresponding elements of and and then squaring, and Upsample() is an up-sampling function, which represents up-sampling Corresponding matrix upsampling is a matrix of the size of the first image.

[0199] In some implementable manners, the execution device can perform feature difference score calculation on the third feature maps at multiple scales and the fourth feature maps at multiple scales by using a mean square error (MSE) method and a cosine similarity (cos) method, to obtain a feature difference score map of the first image, but the application is not limited thereto.

[0200] For example, it is assumed that the feature difference of the jth third feature map and the jth fourth feature map in the feature difference score map of the first image is denoted as Wherein, The jth third feature map is denoted as The jth fourth feature map is denoted as The feature difference score map of the first image is denoted as S, and the feature difference score map can be calculated by the following formula (3):

[0201] Wherein, Indicates And The corresponding element point product is summed, L represents the number of convolution layers in the feature extraction network, or the number of feature extraction layers, Indicates And The corresponding element subtraction is squared, and Upsample() is an upsampling function, which indicates that Corresponding matrix upsampling is a matrix of the size of the first image.

[0202] FIG. 13 is a schematic diagram of calculation of a feature difference score map provided by an embodiment of the application, as shown in FIG. 13, the execution device can perform feature difference score calculation on the third feature maps at multiple scales and the fourth feature maps at multiple scales by using a mean square error (MSE) method and a cosine similarity (cos) method, to obtain a feature difference score map of the first image.

[0203] It should be understood that the application does not limit the calculation method of the feature difference score map.

[0204] In some implementable manners, S350 can include:

[0205] S350-1B: calculating the absolute value of the difference between the first multi-scale feature sequence and the third reconstructed feature sequence output by the Nth layer feature reconstruction network, to obtain a feature difference score map.

[0206] Wherein, each score in the feature difference score map corresponds to a feature, which corresponds to a region or position of the first image, and the region or position includes one or more pixel points.

[0207] The S360 will be described below:

[0208] In some implementable manners, the executing device can determine, as the abnormal pixel point in the first image, a pixel point in the feature difference score map whose score is higher than a preset score.

[0209] In some implementable manners, the preset score can be 0.7, 0.8, 0.9, etc., and the embodiments of the present application do not limit this.

[0210] In some implementable manners, the executing device can determine, as the abnormal pixel point in the first image, a pixel point in the feature difference score map whose product of the score and a preset factor is higher than a preset score.

[0211] In some implementable manners, the preset factor can be any real number greater than 0 and less than 1, and the embodiments of the present application do not limit this.

[0212] It should be understood that the embodiments of the present application do not limit the detection manner of the abnormal pixel point.

[0213] The abnormal detection method provided by the embodiments of the present application will be exemplarily described below in combination with FIG. 14:

[0214] Assume that a model composed of a feature extraction network and an N-layer feature reconstruction network is referred to as a target model, FIG. 14 is a schematic diagram of another target model provided by an embodiment of the present application, as shown in FIG. 14, the execution device can perform multi-scale feature extraction on the image through the pre-trained model in the feature extraction network to obtain feature maps under multiple scales. Further, the feature maps under multiple scales are processed to obtain a first multi-scale feature sequence. Each layer of the feature extraction network in the N-layer feature reconstruction network includes a first attention network and a second attention network, a summation and normalization layer, a feedforward network, another summation and normalization layer. Among them, the execution device can convert each feature in the input feature sequence in the i-th layer of the feature reconstruction network into a first query vector and a first value vector through the first attention network, and convert each feature in the reference feature sequence in the i-th layer of the feature reconstruction network into a first key vector. For each feature in the input feature sequence, the execution device performs scaled dot product on the transpose of each first key vector and the first query vector corresponding to the feature through the first attention network to obtain a third attention matrix corresponding to the feature, calculates the sum of the third attention matrix corresponding to the feature and the first mask matrix corresponding to the feature, and performs normalization processing to obtain a first attention matrix corresponding to the feature; further, the execution device can calculate the product of the first attention matrix corresponding to the feature and the matrix composed of all first value vectors through the first attention network to obtain the reconstruction feature corresponding to the feature, and the reconstruction feature corresponding to each feature in the input feature sequence constitutes a first reconstruction feature sequence. Similarly, the execution device can convert each feature in the input feature sequence in the i-th layer of the feature reconstruction network into a second query vector through the second attention network, and convert each feature in the reference feature sequence in the i-th layer of the feature reconstruction network into a second key vector and a second value vector. For each feature in the input feature sequence, the execution device performs scaled dot product on the transpose of each second key vector and the second query vector corresponding to the feature through the second attention network to obtain a fourth attention matrix corresponding to the feature; calculate the sum of the fourth attention matrix corresponding to the feature and the second mask matrix corresponding to the feature, and perform normalization processing to obtain a second attention matrix corresponding to the feature; further, the execution device can calculate the product of the second attention matrix corresponding to the feature and the matrix composed of all second value vectors through the second attention network to obtain the reconstruction feature corresponding to the feature, and the reconstruction feature corresponding to each feature in the input feature sequence constitutes a first reconstruction feature sequence. Further, the summation and normalization layer performs weighted summation on the two first reconstruction feature sequences and performs normalization processing to obtain a second reconstruction feature sequence. Finally, the execution device performs linear and nonlinear transformation on the second reconstruction feature sequence through the feedforward network layer to obtain a transformation result, and then performs summation on the transformation result and the second reconstruction feature sequence through the summation and normalization layer, and performs normalization processing on the summation result to obtain a third reconstruction feature sequence.

[0215] The embodiment of the present application provides an anomaly detection method, comprising: performing multi-scale feature processing on a first image through a feature extraction network to obtain a first multi-scale feature sequence; performing reconstruction on an input feature sequence of an i-th layer feature reconstruction network based on a reference feature sequence input into the i-th layer feature reconstruction network through each attention network in M attention networks in the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the first multi-scale feature sequence, i=2...N, N and M are positive integers, and the reference feature sequence conforms to a normal image feature mode; if M is equal to 1, performing normalization processing on the first reconstructed feature sequence through a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, performing weighted summation on M first reconstructed feature sequences through the sum and normalization layer, and performing normalization processing on a weighted summation result to obtain the second reconstructed feature sequence; performing processing on the second reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence; determining a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by an N-th layer feature reconstruction network; and determining an abnormal pixel point in the first image based on the feature difference score map of the first image. Since no residual connection is included on the attention network of the feature extraction network, the possibility of learning shortcuts can be reduced, and the reference feature sequence can be introduced in each layer feature reconstruction network, so that the similarity between the abnormal features in the input feature sequence and the reference feature sequence is small, and the similarity between the normal features in the input feature sequence and the reference feature sequence is large, so that the first attention matrix corresponding to each feature in the input feature sequence can reflect the difference between the feature and different reference features, thereby reducing the possibility of rank collapse of the attention matrix, and further improving the reliability of anomaly detection.

[0216] In addition, the anomaly detection method provided by the embodiment of the present application can be applied to multi-class anomaly detection and single-class anomaly detection, and the embodiment of the present application does not limit this.

[0217] FIG. 15 is a flowchart of a model training method provided by an embodiment of the present application. The method can be executed by an electronic device, which can be the training device 203 in FIG. 2, or the electronic device 2100 as shown in FIG. 21, but is not limited thereto. In the embodiment, the training device is taken as an example for description. As shown in FIG. 15, the method can comprise:

[0218] S1510: performing multi-scale feature processing on the second image by the feature extraction network to obtain a second multi-scale feature sequence;

[0219] S1520: reconstructing, by each of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on the reference feature sequence input into the i-th feature reconstruction network, to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the second multi-scale feature sequence, i=2…N, and N and M are positive integers;

[0220] S1530: if M is equal to 1, performing normalization processing on the fourth reconstructed feature sequence by a summing and normalization layer to obtain a fifth reconstructed feature sequence; or if M is greater than 1, performing weighted summing on the M fourth reconstructed feature sequences by the summing and normalization layer, and performing normalization processing on the weighted summing result to obtain the fifth reconstructed feature sequence;

[0221] S1540: processing the fifth reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a sixth reconstructed feature sequence;

[0222] S1550: calculating a loss of a target model composed of the feature extraction network and the N feature reconstruction networks based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th feature reconstruction network;

[0223] S1560: training the target model and the reference feature sequence of each layer in the N feature reconstruction networks based on the loss of the target model.

[0224] The S1510 is described in detail as follows:

[0225] In some implementable manners, the execution device can capture the to-be-detected object by an image acquisition device to obtain the second image. The image acquisition device can be an image acquisition unit, such as a camera, in the execution device, or a peripheral device of the execution device, and the present application does not limit the image acquisition device.

[0226] It should be understood that the second image is a positive sample.

[0227] It should be understood that the S1510 can be implemented by any of the following implementable manners, but is not limited thereto:

[0228] Implementable manner one, the S1510 can include:

[0229] S1510-1A: performing multi-scale feature extraction on the second image by a pre-trained model in the feature extraction network to obtain a fifth feature map under a multi-scale;

[0230] It should be understood that the explanation of the pre-trained model can refer to the above, and the embodiments of the present application will not be repeated.

[0231] It should be understood that the explanation of the fifth feature map under a multi-scale can refer to the explanation of the first feature map under a multi-scale, and the embodiments of the present application will not be repeated.

[0232] S1510-2A: fusing the fifth feature map under a multi-scale by a fusion layer in the feature extraction network to obtain a second fusion feature map;

[0233] It should be understood that the fusion of the fifth feature map under a multi-scale can enhance the image features, and thus the robustness of the anomaly detection can be improved.

[0234] In some implementable manners, the training device can perform an average pooling operation on the fifth feature map under a multi-scale by the fusion layer to obtain a plurality of sixth feature maps under the same scale; and perform merging in the channel dimension on the plurality of sixth feature maps under the same scale by the fusion layer to obtain the second fusion feature map.

[0235] In some implementable manners, the execution device can perform padding or cropping operation on the fifth feature map under a multi-scale by the fusion layer to obtain a plurality of sixth feature maps under the same scale; and perform merging in the channel dimension on the plurality of sixth feature maps under the same scale by the fusion layer to obtain the second fusion feature map.

[0236] It should be understood that the average pooling operation and the merging manner of the feature map can refer to the above, and the embodiments of the present application will not be repeated.

[0237] S1510-3A: performing shape conversion on the second fusion feature map by a conversion layer in the feature extraction network to obtain a third multi-scale feature sequence, and obtaining a second multi-scale feature sequence based on the third multi-scale feature sequence.

[0238] It should be understood that the shape conversion manner of the second fusion feature map can refer to the shape conversion manner of the first fusion feature map, and the embodiments of the present application will not be repeated.

[0239] In some implementable manners, the training device can perform perturbation operation on the third multi-scale feature sequence by the conversion layer to obtain the second multi-scale feature sequence.

[0240] In some implementable manners, the training device can randomly sample a noise token sequence from a Gaussian distribution, and sum the third scale feature sequence and the noise token sequence to obtain the second multi-scale feature sequence, but the application is not limited thereto.

[0241] FIG. 16 is a schematic diagram of another feature extraction network provided by the embodiments of the application. As shown in FIG. 16, after the sixth feature map under the multi-scale is processed, a third multi-scale feature sequence is formed, and then the third scale feature sequence is summed with the noise token sequence to obtain a second multi-scale feature sequence, wherein each box in the second multi-scale feature sequence represents a D-dimensional vector.

[0242] It should be understood that, by performing the perturbation operation on the third multi-scale feature sequence, the output of the attention network in the feature reconstruction network includes the perturbed multi-scale feature sequence, i.e., the second multi-scale feature sequence, instead of directly including the third multi-scale feature sequence, so that the possibility of learning shortcuts can be reduced, and thus the generalization ability and robustness of the target model can be improved.

[0243] In the second implementable manner, S1510 can include:

[0244] S1510-1B: inputting the second image and the size requirement to be met by the third multi-scale feature sequence into the neural network model to obtain the third multi-scale feature sequence;

[0245] S1510-2B: obtaining the second multi-scale feature sequence based on the third multi-scale feature sequence.

[0246] It should be understood that the explanation and description of S1510-1B can refer to the explanation and description of S310-1B, and the explanation and description of S1510-2B can refer to the explanation and description of how to obtain the second multi-scale feature sequence based on the third multi-scale feature sequence introduced above, and the embodiments of the application will not be repeated here.

[0247] S1520 will be described in detail as follows:

[0248] In some implementable manners, the M attention networks include at least one of the following: the first attention network and the second attention network; correspondingly, by each attention network in the M attention networks in the i-th layer feature reconstruction network, the input feature sequence of the i-th layer feature reconstruction network is reconstructed based on the reference feature sequence input into the i-th layer feature reconstruction network to obtain a fourth reconstructed feature sequence, including at least one of the following:

[0249] 1. Through the first attention network, each feature in the input feature sequence in the i-th layer feature reconstruction network is converted into a third query vector and a third value vector, and each feature in the reference feature sequence in the i-th layer feature reconstruction network is converted into a third key vector; based on the third query vector and the third key vector, a fifth attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated; and based on the fifth attention matrix and the third value vector, a fourth reconstructed feature sequence is calculated.

[0250] 2. Through the second attention network, each feature in the input feature sequence in the i-th layer feature reconstruction network is converted into a fourth query vector, and each feature in the reference feature sequence in the i-th layer feature reconstruction network is converted into a fourth key vector and a fourth value vector; based on the fourth query vector and the fourth key vector, a sixth attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated; and based on the sixth attention matrix and the fourth value vector, a fourth reconstructed feature sequence is calculated.

[0251] The following describes the case of obtaining the fourth reconstructed feature sequence through the first attention network:

[0252] It should be understood that since the third key vector is converted from the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature pattern, so that when the fifth attention matrix is calculated based on the third key vector and the third query vector, the similarity between the abnormal feature in the input feature sequence, i.e., the feature corresponding to the abnormal region, and the third key vector is small, therefore, the fourth reconstructed feature sequence usually does not retain the abnormal image feature pattern, and the probability of retaining the normal image feature pattern is high, so that the difference between the input feature sequence and the reconstructed feature sequence is large, which is further conducive to anomaly detection.

[0253] Suppose each feature in the input feature sequence in the i-th layer feature reconstruction network is called a third feature, then the execution device obtains the fifth attention matrix corresponding to the third feature through the first attention network in the following any implementable manner, but is not limited thereto:

[0254] In some implementable manners, the execution device performs scaling dot product on the transpose of each third key vector and the third query vector corresponding to the third feature through the first attention network, and performs normalization processing to obtain the fifth attention matrix corresponding to the third feature.

[0255] In some implementable manners, the execution device performs scaling point multiplication on the third query vector corresponding to the third feature and the transpose of each third key vector through the first attention network to obtain a seventh attention matrix corresponding to the third feature; and calculates the sum of the seventh attention matrix corresponding to the third feature and a third mask matrix corresponding to the third feature, and performs normalization processing to obtain a fifth attention matrix corresponding to the third feature; wherein the values of the third feature corresponding position and the surrounding positions in the third mask matrix are negative infinity, and the values of the other positions are 0.

[0256] The third mask matrix can also be referred to as a neighbor masked attention matrix or an attention map, which is used to mask the attention values corresponding to the third feature and its neighbor features in the seventh attention matrix corresponding to the third feature.

[0257] In some implementable manners, the neighbor features of the third feature can include at least one of, but are not limited to, the upper neighbor feature, the lower neighbor feature, the left neighbor feature, the right neighbor feature, the upper left neighbor feature, the lower left neighbor feature, the upper right neighbor feature, and the lower right neighbor feature of the third feature.

[0258] It should be understood that the surrounding positions of the third feature corresponding position refer to the corresponding positions of the neighbor features of the third feature in the third mask matrix.

[0259] It should be understood that the problem of learning shortcut is mainly caused by the fact that the output of the network directly includes the input of the network. Based on this, the embodiments of the present application propose to mask the attention values corresponding to the third feature and its neighbor features in the seventh attention matrix corresponding to the third feature, so that when reconstructing the third feature, the third value vector corresponding to the third feature and its neighbor features can be ignored, thereby preventing the reconstructed feature from including the third feature and its neighbor features, reducing the possibility of learning shortcut, and further improving the robustness of the anomaly detection method.

[0260] The following describes the case of obtaining the fourth reconstructed feature sequence through the second attention network:

[0261] It should be understood that since the fourth key vector is derived from features in the reference feature sequence, and the reference feature sequence conforms to normal image feature patterns, when calculating the sixth attention matrix based on the fourth key vector and the fourth query vector, the anomalous features in the input feature sequence—that is, the features corresponding to the anomalous regions—have low similarity to the fourth key vector. Therefore, the fourth reconstructed feature sequence usually does not retain anomalous image feature patterns, but rather retains normal image feature patterns, resulting in a larger difference between the input feature sequence and the reconstructed feature sequence, which is beneficial for anomaly detection. Furthermore, since the fourth value vector is also derived from features in the reference feature sequence, and the reference feature sequence conforms to normal image feature patterns, the generated fourth reconstructed feature sequence based on the fourth value vector is more likely to retain normal image feature patterns, further increasing the difference between the input feature sequence and the reconstructed feature sequence, and thus further improving the robustness of the anomaly detection method.

[0262] Suppose that each feature in the input feature sequence of the i-th layer feature reconstruction network is called the fourth feature. Then, the execution device can obtain the sixth attention matrix corresponding to the fourth feature through the second attention network in any of the following possible ways, but is not limited to:

[0263] In some implementations, the execution device uses a second attention network to perform a scaled dot product between the fourth query vector corresponding to the fourth feature and the transpose of each fourth key vector, and then performs normalization to obtain the sixth attention matrix corresponding to the fourth feature.

[0264] In some implementations, the execution device uses a second attention network to perform a scaled dot product between the fourth query vector corresponding to the fourth feature and the transpose of each fourth key vector to obtain the eighth attention matrix corresponding to the fourth feature; and calculates the sum of the eighth attention matrix corresponding to the fourth feature and the fourth mask matrix corresponding to the fourth feature, and performs normalization processing to obtain the sixth attention matrix corresponding to the fourth feature; wherein, the value of the position corresponding to the fourth feature and the surrounding positions in the fourth mask matrix is ​​0, and the value of the remaining positions is negative infinity.

[0265] The fourth mask matrix is ​​used to mask the attention values ​​of the other features in the eighth attention matrix corresponding to the fourth feature, excluding the fourth feature and its neighboring features.

[0266] In some possible implementations, the neighbor features of the fourth feature may include at least one of the following, but not limited to: the neighbor features directly above the fourth feature, the neighbor features directly below the fourth feature, the neighbor features to the left of the fourth feature, the neighbor features to the right of the fourth feature, the neighbor features to the top left of the fourth feature, the neighbor features to the bottom left of the fourth feature, the neighbor features to the top right of the fourth feature, and the neighbor features to the bottom right of the fourth feature.

[0267] It should be understood that the surrounding positions of the fourth feature corresponding position refer to the corresponding positions of the neighbor features of the fourth feature in the fourth mask matrix.

[0268] It should be understood that, since the fourth value vector is the result of the conversion of the features in the reference feature sequence, and the reference feature sequence conforms to the normal image feature mode, the embodiments of the present application propose that the attention values corresponding to the features other than the fourth feature and its neighbor features in the eighth attention matrix corresponding to the fourth feature can be masked, so that when reconstructing the fourth feature, the fourth value vector corresponding to the fourth feature and its neighbor features can be mainly referred to, thereby further improving the robustness of the anomaly detection method.

[0269] It should be understood that, in the case of the combination of the first attention network and the second attention network, or in the case of the combination of the first attention network based on the MLKA mechanism and the first attention network based on the LCA mechanism, for each feature in the input feature sequence, the first attention network can reconstruct the feature to obtain a reconstructed feature, and similarly, the second attention network can also reconstruct the feature to obtain a reconstructed feature, and the summing and normalization layer can perform weighted summing on the two reconstructed features and then perform normalization processing to obtain a reconstructed feature corresponding to the feature.

[0270] It should be noted that the third feature can be each feature of the input feature sequence, and the fourth feature can also be each feature of the input feature sequence. The third feature and the fourth feature can be the same feature or different features. It is worth mentioning that, in the case of the combination of the first attention network and the second attention network, or in the case of the combination of the first attention network based on the MLKA mechanism and the first attention network based on the LCA mechanism, the third feature and the fourth feature refer to the same feature.

[0271] The S330 will be described in detail below:

[0272] In some implementable manners, if M is equal to 1, the execution device performs normalization processing on the fourth reconstructed feature sequence through the summing and normalization layer to obtain a fifth reconstructed feature sequence.

[0273] In some implementable manners, if M is greater than 1, the execution device performs weighted summing on the M fourth reconstructed feature sequences through the summing and normalization layer to obtain a weighted summing result; and performs normalization processing on the weighted summing result to obtain a fifth reconstructed feature sequence.

[0274] The S1540 will be described in detail below:

[0275] In some implementable manners, the other network layer in the i-th layer feature reconstruction network comprises a feedforward network layer and another summation and normalization layer. Based on this, the training device processes the fifth reconstructed feature sequence through the other network layer in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence, which comprises that the training device performs linear and nonlinear transformation on the fifth reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain a second transformation result, and then performs summation on the second transformation result and the fifth reconstructed feature sequence through the summation and normalization layer, and performs normalization processing on the summation result to obtain the sixth reconstructed feature sequence.

[0276] In some implementable manners, the other network layer in the i-th layer feature reconstruction network comprises a feedforward network layer and another summation and normalization layer. Based on this, the training device processes the fifth reconstructed feature sequence through the other network layer in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence, which comprises that the training device performs linear and nonlinear transformation on the fifth reconstructed feature sequence through the feedforward network layer in the i-th layer feature reconstruction network to obtain a second transformation result, and then performs summation on the second transformation result and the fifth reconstructed feature sequence through the summation and normalization layer, and performs normalization processing on the summation result to obtain the sixth reconstructed feature sequence.

[0277] It should be understood that the embodiments of the present application do not limit the other network layer in the i-th layer feature reconstruction network, and do not limit the corresponding processing of the other network layer.

[0278] The S1550 will be described below:

[0279] In some implementable manners, the training device can restore the sixth reconstructed feature sequence output by the N-th layer feature reconstruction network to a seventh feature map under a plurality of scales; restore the second multi-scale feature sequence to an eighth feature map under a plurality of scales; and calculate the loss of the target model based on the seventh feature map under a plurality of scales and the eighth feature map under a plurality of scales.

[0280] It should be understood that the restoration of the reconstructed feature sequence to the feature map under a plurality of scales can be referred to the foregoing, and the embodiments of the present application will not be described again.

[0281] In some implementable manners, the training device can calculate the loss of the target model under each scale based on the seventh feature map and the eighth feature map under each scale; and sum the loss of the target model under each scale to obtain the total loss of the target model.

[0282] In some implementable manners, the training device can calculate the loss of the target model under each scale based on the seventh feature map and the eighth feature map under each scale by using a cosine similarity (Cos) method.

[0283] For example, it is assumed that the cosine loss between the j-th seventh feature map and the j-th eighth feature map is denoted as This can be calculated using the following formula (4). Let the j-th seventh feature map be denoted as... Let the j-th eighth feature map be denoted as Let H be the height and width of the seventh and eighth feature maps, respectively. j and W j Let the total loss be denoted as L. Then the total loss L can be calculated using the following formula (5):

[0284] in, express and The corresponding element-wise dot product is then summed.

[0285] In some feasible approaches, the training device can utilize mean squared error (MSE) to calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale.

[0286] For example, suppose the j-th seventh feature map is denoted as Let the j-th eighth feature map be denoted as Let H be the height and width of the seventh and eighth feature maps, respectively. j and W j Let the total loss be denoted as L. Then the total loss L can be calculated using the following formula (6):

[0287] In some feasible approaches, the training device can use cosine similarity (Cos) and mean squared error (MSE) to calculate the loss of the target model at each scale based on the seventh and eighth feature maps at each scale.

[0288] For example, suppose the cosine loss between the j-th seventh feature map and the j-th eighth feature map is denoted as... in, It can be calculated using formula (7). Let the j-th seventh feature map be denoted as... Let the j-th eighth feature map be denoted as Let H be the height and width of the seventh and eighth feature maps, respectively. j and W j Let the total loss be denoted as L. Then the total loss L can be calculated using the following formula (8):

[0289] The explanations of the symbols in these two formulas can be found above, and will not be repeated in this embodiment.

[0290] FIG. 17 is a schematic diagram of calculation of the target model loss according to an embodiment of the present application. As shown in FIG. 17, the training device can use the mean square error (MSE) method and the cosine similarity (Cos) method to calculate the loss of the seventh feature map in the multi-scale and the eighth feature map in the multi-scale, and obtain the total loss of the target model.

[0291] It should be understood that the embodiments of the present application do not limit the calculation method of the target model loss.

[0292] The S1560 will be described below.

[0293] In some possible implementations, the training device can adjust the parameters of the target model based on the total loss of the target model, and stop the training when the number of training reaches a preset number or the total loss reaches a minimum value.

[0294] It should be noted that before training the target model, an initial value can be assigned to the reference feature sequence of each layer of the N-layer feature reconstruction network, and during training the target model, the reference feature sequence of each layer can be adjusted at the same time. Since the present application does not include a residual connection on the attention network of the feature extraction network, the possibility of learning shortcuts can be reduced. Based on this, during training the target model, the reference feature sequence can learn the normal image feature mode.

[0295] The abnormality detection method according to an embodiment of the present application will be described below with reference to FIG. 18.

[0296] Assume that a model composed of a feature extraction network and an N-layer feature reconstruction network is referred to as a target model, FIG. 18 is a schematic diagram of another target model provided by an embodiment of the present application, as shown in FIG. 18, the training device can perform multi-scale feature extraction on the image through the pre-trained model in the feature extraction network to obtain feature maps at multiple scales. Further, the feature maps at multiple scales are processed to obtain a third multi-scale feature sequence, and the third multi-scale feature sequence and the noise label sequence are summed to obtain a second multi-scale feature sequence. Each layer of the feature extraction network in the N-layer feature reconstruction network includes: a first attention network and a second attention network, a sum and normalization layer, a feedforward network, and another sum and normalization layer. Among them, the training device can convert each feature in the input feature sequence in the i-th layer of the feature reconstruction network into a third query vector and a third value vector through the first attention network, and convert each feature in the reference feature sequence in the i-th layer of the feature reconstruction network into a third key vector. For each feature in the input feature sequence, the training device performs scaled dot product on the transpose of each third key vector and the third query vector corresponding to the feature through the first attention network to obtain a seventh attention matrix corresponding to the feature, calculates the sum of the seventh attention matrix corresponding to the feature and the third mask matrix corresponding to the feature, and performs normalization processing to obtain a fifth attention matrix corresponding to the feature; further, the training device can calculate the product of the fifth attention matrix corresponding to the feature and the matrix composed of all third value vectors through the first attention network to obtain the reconstruction feature corresponding to the feature, and the reconstruction feature corresponding to each feature in the input feature sequence constitutes a fourth reconstruction feature sequence. Similarly, the training device can convert each feature in the input feature sequence in the i-th layer of the feature reconstruction network into a fourth query vector through the second attention network, and convert each feature in the reference feature sequence in the i-th layer of the feature reconstruction network into a fourth key vector and a fourth value vector. For each feature in the input feature sequence, the training device performs scaled dot product on the transpose of each fourth key vector and the fourth query vector corresponding to the feature through the second attention network to obtain an eighth attention matrix corresponding to the feature; calculate the sum of the eighth attention matrix corresponding to the feature and the fourth mask matrix corresponding to the feature, and perform normalization processing to obtain a sixth attention matrix corresponding to the feature; further, the training device can calculate the product of the sixth attention matrix corresponding to the feature and the matrix composed of all fourth value vectors through the second attention network to obtain the reconstruction feature corresponding to the feature, and the reconstruction feature corresponding to each feature in the input feature sequence constitutes a fourth reconstruction feature sequence. Further, the sum and normalization layer performs weighted sum on the two fourth reconstruction feature sequences and performs normalization processing to obtain a fifth reconstruction feature sequence.Finally, the training device performs linear and nonlinear transformation on the fifth reconstructed feature sequence through the feedforward network layer, obtains a transformation result, and then performs summation and normalization on the transformation result and the fifth reconstructed feature sequence through the summation and normalization layer, and performs normalization processing on the summation result, to obtain a sixth reconstructed feature sequence.

[0297] The embodiment of the application provides a model training method, which comprises: performing multi-scale feature processing on a second image through a feature extraction network to obtain a second multi-scale feature sequence; reconstructing an input feature sequence of an i-th layer feature reconstruction network based on a reference feature sequence input into the i-th layer feature reconstruction network through each attention network in M attention networks in the i-th layer feature reconstruction network to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the second multi-scale feature sequence, i=2...N, and N and M are positive integers; if M is equal to 1, performing normalization processing on the fourth reconstructed feature sequence through a summation and normalization layer to obtain a fifth reconstructed feature sequence; or if M is greater than 1, performing weighted summation on M fourth reconstructed feature sequences through the summation and normalization layer, and performing normalization processing on the weighted summation result to obtain the fifth reconstructed feature sequence; processing the fifth reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence; calculating a loss of a target model composed of the feature extraction network and N layer feature reconstruction networks based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th layer feature reconstruction network; and training the target model and the reference feature sequence of each layer in the N layer feature reconstruction network based on the loss of the target model. Since no residual connection is included on the attention network of the feature extraction network, the possibility of learning shortcut can be reduced, so that the reference feature sequence of each layer and the target model can learn normal image feature patterns, so that the input feature sequence can be reconstructed into a normal feature sequence with a high probability when the feature reconstruction is performed based on the target model and the reference feature sequence, thereby making the difference between the input feature sequence with abnormal features and the reconstructed feature sequence larger, and further being beneficial to anomaly detection.

[0298] The preferred embodiments of the present application are described in detail above with reference to the drawings, but the present application is not limited to the specific details of the above-described embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solutions of the present application, and these simple modifications all belong to the protection scope of the present application. For example, in the above-described specific embodiments, various specific technical features described can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combination manners are not described again in the present application. For another example, various different embodiments of the present application can also be combined in any manner, as long as it does not deviate from the idea of the present application, and it should also be considered as disclosed content of the present application.

[0299] It should also be understood that, in various method embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0300] The above describes the method provided by the embodiments of the present application, and the following describes the abnormality detection device and model training device provided by the embodiments of the present application.

[0301] FIG. 19 is a schematic diagram of an abnormality detection device 1900 provided by an embodiment of the present application, which includes:

[0302] The multi-scale feature processing module 1910 is configured to perform multi-scale feature processing on the first image by the feature extraction network to obtain a first multi-scale feature sequence.

[0303] The first feature reconstruction module 1920 is configured to reconstruct, by each attention network of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on a reference feature sequence input into the i-th feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i=2…N, and N and M are positive integers, and the reference feature sequence conforms to a normal image feature mode.

[0304] The second feature reconstruction module 1930 is configured to, if M is equal to 1, perform normalization processing on the first reconstructed feature sequence by a summation and normalization layer to obtain a second reconstructed feature sequence; or, if M is greater than 1, perform weighted summation on the M first reconstructed feature sequences by the summation and normalization layer, and perform normalization processing on the weighted summation result to obtain the second reconstructed feature sequence.

[0305] The third feature reconstruction module 1940 is configured to process the second reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence.

[0306] The first determination module 1950 is configured to determine a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th layer feature reconstruction network.

[0307] The second determination module 1960 is configured to determine an abnormal pixel point in the first image based on the feature difference score map of the first image.

[0308] In some implementations, the M attention networks include at least one of the following: a first attention network and a second attention network; correspondingly, the first feature reconstruction module 1920 is configured to perform at least one of the following:

[0309] The first attention network is configured to convert each feature in the input feature sequence in the i-th layer feature reconstruction network into a first query vector and a first value vector, and convert each feature in the reference feature sequence in the i-th layer feature reconstruction network into a first key vector; based on the first query vector and the first key vector, calculate a first attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network; and based on the first attention matrix and the first value vector, calculate the first reconstructed feature sequence.

[0310] The second attention network is configured to convert each feature in the input feature sequence in the i-th layer feature reconstruction network into a second query vector, and convert each feature in the reference feature sequence in the i-th layer feature reconstruction network into a second key vector and a second value vector; based on the second query vector and the second key vector, calculate a second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network; and based on the second attention matrix and the second value vector, calculate the first reconstructed feature sequence.

[0311] In some implementations, the first feature reconstruction module 1920 is specifically configured to: perform scaled dot product on the first query vector corresponding to the first feature and the transpose of each first key vector to obtain a third attention matrix corresponding to the first feature; and calculate the sum of the third attention matrix corresponding to the first feature and a first mask matrix corresponding to the first feature, and perform normalization processing to obtain the first attention matrix corresponding to the first feature; wherein the values of the first feature corresponding position and the surrounding positions in the first mask matrix are negative infinity, and the values of the other positions are 0.

[0312] In some implementable manners, the first feature reconstruction module 1920 is specifically configured to: perform scaled dot products between the second query vector corresponding to the second feature and the transposes of each second key vector to obtain a fourth attention matrix corresponding to the second feature; wherein the second feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; calculate a sum of the fourth attention matrix corresponding to the second feature and a second mask matrix corresponding to the second feature, and perform normalization processing to obtain a second attention matrix corresponding to the second feature; wherein the values of the second feature corresponding positions and the surrounding positions in the second mask matrix are 0, and the values of the remaining positions are negative infinity.

[0313] In some implementable manners, when performing weighted sum of the first reconstructed feature sequence output by the first attention network and the first reconstructed feature sequence output by the second attention network through the sum and normalization layer, the weight corresponding to the first reconstructed feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstructed feature sequence output by the second attention network is a hyperparameter greater than 1.

[0314] In some implementable manners, the multi-scale feature processing module 1910 is specifically configured to: perform multi-scale feature extraction on the first image through a pre-trained model in the feature extraction network to obtain a first feature map under multiple scales; perform fusion on the first feature map under multiple scales through a fusion layer in the feature extraction network to obtain a first fused feature map; and perform shape conversion on the first fused feature map through a conversion layer in the feature extraction network to obtain a first multi-scale feature sequence.

[0315] In some implementable manners, the multi-scale feature processing module 1910 is specifically configured to: perform average pooling operations on the first feature map under multiple scales through the fusion layer respectively to obtain multiple second feature maps under the same scale; and perform merging in the channel dimension on the multiple second feature maps under the same scale through the fusion layer to obtain the first fused feature map.

[0316] In some implementable manners, the first determination module 1950 is specifically configured to: restore the third reconstructed feature sequence output by the N-th layer feature reconstruction network to a third feature map under multiple scales; restore the first multi-scale feature sequence to a fourth feature map under multiple scales; and perform feature difference score calculation on the third feature map under multiple scales and the fourth feature map under multiple scales to obtain a feature difference score map of the first image.

[0317] In some implementable manners, the second determination module 1960 is specifically configured to: determine the pixel points with scores higher than a preset score in the feature difference score map as the abnormal pixel points in the first image.

[0318] It should be understood that the apparatus embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, the foregoing and other operations and / or functions of the various modules in the apparatus 1900 are not described here. Specifically, the apparatus 1900 shown in FIG. 19 can perform the corresponding method embodiments of FIG. 3, and the foregoing and other operations and / or functions of the various modules in the apparatus 1900 are respectively for implementing the corresponding flows in the various methods in FIG. 3, which are not described here for brevity.

[0319] The apparatus 1900 of the embodiments of the present application is described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution, or executed by a combination of hardware and software modules in the code processing. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory to complete the steps in the above method embodiments in combination with the hardware thereof.

[0320] FIG. 20 is a schematic diagram of a model training apparatus 2000 provided by the embodiments of the present application. As shown in FIG. 20, the apparatus 2000 comprises:

[0321] The multi-scale feature processing module 2010 is configured to perform multi-scale feature processing on the second image by using the feature extraction network to obtain a second multi-scale feature sequence.

[0322] The first feature reconstruction module 2020 is configured to reconstruct the input feature sequence of the i-th feature reconstruction network based on the reference feature sequence input into the i-th feature reconstruction network by using each attention network in the M attention networks in the i-th feature reconstruction network to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is the output feature sequence of the (i-1)-th feature reconstruction network, the input feature sequence of the first feature reconstruction network is the second multi-scale feature sequence, i=2…N, and N and M are positive integers.

[0323] The second feature reconstruction module 2030 is configured to, if M is equal to 1, perform normalization processing on the fourth reconstructed feature sequence by using a summing and normalization layer to obtain a fifth reconstructed feature sequence; or, if M is greater than 1, perform weighted summation on the M fourth reconstructed feature sequences by using a summing and normalization layer, and perform normalization processing on the weighted summation result to obtain the fifth reconstructed feature sequence.

[0324] The third feature reconstruction module 2040 is configured to process the fifth reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence.

[0325] The loss calculation module 2050 is configured to calculate a loss of the target model composed of the feature extraction network and the N-layer feature reconstruction network based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-layer feature reconstruction network.

[0326] The training module 2060 is configured to train the target model and the reference feature sequence of each layer in the N-layer feature reconstruction network based on the loss of the target model.

[0327] In some implementations, the multi-scale feature processing module 2010 is specifically configured to: perform multi-scale feature extraction on the second image through a pre-trained model in the feature extraction network to obtain a fifth feature map under a multi-scale; perform fusion on the fifth feature map under the multi-scale through a fusion layer in the feature extraction network to obtain a second fused feature map; perform shape conversion on the second fused feature map through a conversion layer in the feature extraction network to obtain a third multi-scale feature sequence, and obtain the second multi-scale feature sequence based on the third multi-scale feature sequence.

[0328] In some implementations, the multi-scale feature processing module 2010 is specifically configured to: perform an average pooling operation on the fifth feature map under the multi-scale through the fusion layer respectively to obtain a plurality of sixth feature maps under the same scale; and perform merging in the channel dimension on the plurality of sixth feature maps under the same scale through the fusion layer to obtain the second fused feature map.

[0329] In some implementations, the multi-scale feature processing module 2010 is specifically configured to: perform a perturbation operation on the third multi-scale feature sequence to obtain the second multi-scale feature sequence.

[0330] In some implementations, the loss calculation module 2050 is specifically configured to: restore the sixth reconstructed feature sequence output by the N-layer feature reconstruction network to a seventh feature map under a multi-scale; restore the second multi-scale feature sequence to an eighth feature map under a multi-scale; and calculate the loss of the target model based on the seventh feature map under the multi-scale and the eighth feature map under the multi-scale.

[0331] In some implementations, the loss calculation module 2050 is specifically configured to: calculate the loss of the target model under each scale based on the seventh feature map and the eighth feature map under each scale; and sum the loss of the target model under each scale to obtain a total loss of the target model.

[0332] It should be understood that the apparatus embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, the foregoing and other operations and / or functions of the various modules in the apparatus 2000 are not described here. Specifically, the apparatus 2000 shown in FIG. 20 can perform the corresponding method embodiments of FIG. 15, and the foregoing and other operations and / or functions of the various modules in the apparatus 2000 are respectively for realizing the corresponding flows in the various methods in FIG. 15, and are not described here for brevity.

[0333] The apparatus 2000 of the embodiments of the present application is described above in the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be realized in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution by the processor, or executed by a combination of hardware and software modules in the code processing. Alternatively, the software modules can be located in mature storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage media is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in combination with the hardware thereof.

[0334] FIG. 21 is a schematic block diagram of an electronic device 2100 provided by the embodiments of the present application. The electronic device 2100 can be an execution device or a training device, but is not limited thereto. As shown in FIG. 21, the electronic device 2100 can include:

[0335] The memory 2110 and the processor 2120, the memory 2110 is used to store the computer program 2130, and transmit the computer program 2130 to the processor 2120. In other words, the processor 2120 can call and run the computer program 2130 from the memory 2110 to realize the method in the embodiments of the present application.

[0336] For example, the processor 2120 can be used to execute the steps in the above method according to the instructions in the computer program 2130.

[0337] In some embodiments of the present application, the processor 2120 can include but is not limited to:

[0338] A general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0339] In some embodiments of the present application, the memory 2110 includes, but is not limited to:

[0340] volatile memory and / or non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), which is used as the external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0341] In some embodiments of the present application, the computer program 2130 can be divided into one or more modules, which are stored in the memory 2110 and executed by the processor 2120 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 2130 in the electronic device.

[0342] As shown in FIG. 21, the electronic device 2100 can further include:

[0343] a transceiver 2140, which can be connected to the processor 2120 or the memory 2110.

[0344] The processor 2120 can control the transceiver 2140 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 2140 can include a transmitter and a receiver. The transceiver 2140 can further include an antenna, and the number of antennas can be one or more.

[0345] It should be understood that various components in the electronic device 2100 are connected through a bus system, and the bus system includes, in addition to a data bus, a power supply bus, a control bus, and a status signal bus.

[0346] According to an aspect of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program. When the computer program is executed by a computer, the computer is enabled to perform the method of the above-mentioned method embodiments. Alternatively, the embodiments of the present application further provide a computer program product containing instructions. When the instructions are executed by a computer, the computer performs the method of the above-mentioned method embodiments.

[0347] According to another aspect of the present application, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method of the above-mentioned method embodiments.

[0348] In other words, when implemented using software, the functions can be implemented in whole or in part using a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the whole or part of the flow or function according to the embodiments of the present application is generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as digital video disc (DVD)), or semiconductor media (such as solid state disk (SSD)) and the like.

[0349] Those skilled in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0350] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0351] The modules illustrated as separate components may or may not be physically separate, and the components illustrated as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected to implement the embodiments of the present application according to actual needs. For example, the functional modules in various embodiments of the present application can be integrated into one processing module, or each module can be physically present separately, or two or more modules can be integrated into one module.

[0352] The above is merely specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An anomaly detection method characterized by, The method comprises the following steps: performing multi-scale feature processing on the first image through a feature extraction network to obtain a first multi-scale feature sequence; reconstructing, through each of the M attention networks in the i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence input into the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is an output feature sequence of an (i-1)-th layer feature reconstruction network, the input feature sequence of a first layer feature reconstruction network is the first multi-scale feature sequence, i=2……N, N and M are positive integers, and the reference feature sequence conforms to a normal image feature mode; if M is equal to 1, performing normalization processing on the first reconstructed feature sequence through a summing and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, performing weighted summing on the M first reconstructed feature sequences through the summing and normalization layer and performing normalization processing on the weighted summing result to obtain the second reconstructed feature sequence; processing the second reconstructed feature sequence through other network layers in the i-th layer feature reconstruction network to obtain a third reconstructed feature sequence; determining a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by an N-th layer feature reconstruction network; determining an abnormal pixel point in the first image based on the feature difference score map of the first image.

2. The method of claim 1, wherein, The M attention networks comprise at least one of the following: a first attention network and a second attention network; The reconstructing, through each of the M attention networks in the i-th layer feature reconstruction network, an input feature sequence of the i-th layer feature reconstruction network based on a reference feature sequence input into the i-th layer feature reconstruction network to obtain a first reconstructed feature sequence, comprises at least one of the following: The first attention network converts each feature in the input feature sequence in the i-th layer feature reconstruction network into a first query vector and a first value vector, and converts each feature in the reference feature sequence in the i-th layer feature reconstruction network into a first key vector; based on the first query vector and the first key vector, the first attention network calculates a first attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network; and based on the first attention matrix and the first value vector, the first attention network calculates the first reconstructed feature sequence; The second attention network converts each feature in the input feature sequence in the i-th layer feature reconstruction network into a second query vector, and converts each feature in the reference feature sequence in the i-th layer feature reconstruction network into a second key vector and a second value vector; based on the second query vector and the second key vector, the second attention network calculates a second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network; and based on the second attention matrix and the second value vector, the second attention network calculates the first reconstructed feature sequence.

3. The method of claim 2, wherein, The first attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the first query vector and the first key vector, including: The first feature corresponding to the first query vector and the transpose of each first key vector is scaled dot product, to obtain the third attention matrix corresponding to the first feature; wherein the first feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; The sum of the third attention matrix corresponding to the first feature and the first mask matrix corresponding to the first feature is calculated, and normalized processing is performed to obtain the first attention matrix corresponding to the first feature; Wherein, the value of the first feature corresponding position and the surrounding position in the first mask matrix is negative infinity, and the value of the rest position is 0.

4. The method of claim 2, wherein, The second attention matrix corresponding to each feature in the input feature sequence in the i-th layer feature reconstruction network is calculated based on the second query vector and the second key vector, including: The second feature corresponding to the second query vector and the transpose of each second key vector is scaled dot product, to obtain the fourth attention matrix corresponding to the second feature; wherein the second feature is each feature in the input feature sequence in the i-th layer feature reconstruction network; The sum of the fourth attention matrix corresponding to the second feature and the second mask matrix corresponding to the second feature is calculated, and normalized processing is performed to obtain the second attention matrix corresponding to the second feature; Wherein, the value of the second feature corresponding position and the surrounding position in the second mask matrix is 0, and the value of the rest position is negative infinity.

5. The method according to any one of claims 2-4, characterized in that, When the first reconstruction feature sequence output by the first attention network and the first reconstruction feature sequence output by the second attention network are weighted and summed by the sum and normalization layer, the weight corresponding to the first reconstruction feature sequence output by the first attention network is 1, and the weight corresponding to the first reconstruction feature sequence output by the second attention network is a hyperparameter greater than 1.

6. The method according to any one of claims 1 to 4, characterized in that, The first multi-scale feature sequence is obtained by performing multi-scale feature processing on the first image through the feature extraction network, including: The first feature map under multiple scales is obtained by performing multi-scale feature extraction on the first image through the pre-trained model in the feature extraction network; The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network; The first multi-scale feature sequence is obtained by shape conversion of the first fusion feature map through the conversion layer in the feature extraction network.

7. The method of claim 6, wherein, The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network, including: The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network, including: The first fusion feature map is obtained by fusing the first feature map under multiple scales through the fusion layer in the feature extraction network, including:

8. A model training method, comprising: Including: The second image is processed by a feature extraction network to obtain a second multi-scale feature sequence; Each of the M attention networks in the i-th layer feature reconstruction network reconstructs the input feature sequence of the i-th layer feature reconstruction network based on the reference feature sequence input into the i-th layer feature reconstruction network, to obtain a fourth reconstructed feature sequence, wherein the input feature sequence of the i-th layer feature reconstruction network is the output feature sequence of the (i-1)-th layer feature reconstruction network, the input feature sequence of the first layer feature reconstruction network is the second multi-scale feature sequence, i=2……N, and N and M are positive integers; If M is equal to 1, the fourth reconstructed feature sequence is normalized by a sum and normalization layer to obtain a fifth reconstructed feature sequence; or if M is greater than 1, the M fourth reconstructed feature sequences are weighted summed by the sum and normalization layer, and the weighted sum result is normalized to obtain the fifth reconstructed feature sequence; The fifth reconstructed feature sequence is processed by other network layers in the i-th layer feature reconstruction network to obtain a sixth reconstructed feature sequence; Loss of a target model composed of the feature extraction network and the N layer feature reconstruction network is calculated based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th layer feature reconstruction network; The target model and the reference feature sequence of each layer in the N layer feature reconstruction network are trained based on the loss of the target model.

9. The method of claim 8, wherein, The second image is processed by a feature extraction network to obtain a second multi-scale feature sequence, including: The second image is processed by a pre-trained model in the feature extraction network to obtain a fifth feature map under multiple scales; The fifth feature map under multiple scales is fused by a fusion layer in the feature extraction network to obtain a second fused feature map; The second fused feature map is shape-converted by a conversion layer in the feature extraction network to obtain a third multi-scale feature sequence, and the second multi-scale feature sequence is obtained based on the third multi-scale feature sequence.

10. The method of claim 9, wherein, The second multi-scale feature sequence is obtained based on the third multi-scale feature sequence, including: The third multi-scale feature sequence is subjected to a perturbation operation to obtain the second multi-scale feature sequence.

11. An abnormality detection device characterized by comprising: Including: A multi-scale feature processing module is configured to process a first image by a feature extraction network to obtain a first multi-scale feature sequence; The first feature reconstruction module is configured to reconstruct, by each of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on a reference feature sequence input into the i-th feature reconstruction network, to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i=2...N, and N and M are positive integers; and the reference feature sequence conforms to a normal image feature mode. The second feature reconstruction module is configured to, if M is equal to 1, normalize the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, perform weighted summation on the M first reconstructed feature sequences by the sum and normalization layer, and perform normalization on a result of the weighted summation to obtain the second reconstructed feature sequence. The third feature reconstruction module is configured to process the second reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a third reconstructed feature sequence. The first determination module is configured to determine a feature difference score map of the first image based on the first multi-scale feature sequence and the third reconstructed feature sequence output by the N-th feature reconstruction network. The second determination module is configured to determine an abnormal pixel point in the first image based on the feature difference score map of the first image.

12. A model training apparatus, comprising: The multi-scale feature processing module is configured to perform multi-scale feature processing on the second image by a feature extraction network to obtain a second multi-scale feature sequence. The first feature reconstruction module is configured to reconstruct, by each of the M attention networks in the i-th feature reconstruction network, an input feature sequence of the i-th feature reconstruction network based on a reference feature sequence input into the i-th feature reconstruction network, to obtain a first reconstructed feature sequence, wherein the input feature sequence of the i-th feature reconstruction network is an output feature sequence of an (i-1)-th feature reconstruction network, the input feature sequence of a first feature reconstruction network is the first multi-scale feature sequence, i=2...N, and N and M are positive integers; and the reference feature sequence conforms to a normal image feature mode. The second feature reconstruction module is configured to, if M is equal to 1, normalize the first reconstructed feature sequence by a sum and normalization layer to obtain a second reconstructed feature sequence; or if M is greater than 1, perform weighted summation on the M first reconstructed feature sequences by the sum and normalization layer, and perform normalization on a result of the weighted summation to obtain the second reconstructed feature sequence. The third feature reconstruction module is configured to process the second reconstructed feature sequence by other network layers in the i-th feature reconstruction network to obtain a third reconstructed feature sequence. The loss calculation module is configured to calculate a loss of a target model composed of the feature extraction network and the N feature reconstruction networks based on the second multi-scale feature sequence and the sixth reconstructed feature sequence output by the N-th feature reconstruction network. ​ a training module configured to train the target model and the reference feature sequence of each layer in the N-layer feature reconstruction network based on a loss of the target model.

13. An electronic device, comprising: comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the method of any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, a computer program product for storing a computer program, which causes a computer to execute the method of any one of claims 1 to 10.

15. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the method of any one of claims 1 to 10.

Images