Abnormity detection method, device and equipment based on differentiable feature matching

Abstract

The invention provides an anomaly detection method, device and equipment based on differentiable feature matching, and the method comprises the steps: obtaining an initial feature extractor which comprises a feature matching network; obtaining a first training sample set; performing first training on the initial feature extractor according to the first training sample set to obtain a first feature extractor; obtaining a second training sample set; and according to the second training sample set, carrying out second training on the first feature extractor to optimize parameters of the feature matching network and obtain a trained feature extractor, and the method is used for achieving the technical effect of improving the anomaly detection efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to an anomaly detection method, device, and equipment based on differentiable feature matching. Background Art

[0002] With the continuous improvement of the technological level, the public's requirements for the quality of products in various industries are increasing day by day. In the field of industrial applications, anomaly detection plays a crucial role. It is not only an important means to prevent failures but also a key link to ensure product quality. Therefore, anomaly detection methods have become a promising direction.

[0003] In the prior art, the methods of anomaly detection are mainly based on feature matching methods, relying on pre-trained backbone models to complete the anomaly detection task of target objects.

[0004] Since in the prior art, the existing anomaly detection method based on differentiable feature matching overly relies on pre-trained backbone models, with low detection accuracy and poor detection efficiency, there are technical problems of low anomaly detection effectiveness. Summary of the Invention

[0005] This application provides an anomaly detection method, device, and equipment based on differentiable feature matching to achieve the technical effect of improving the acquisition efficiency of digital twin models.

[0006] In a first aspect, this application provides an anomaly detection method based on differentiable feature matching, including:

[0007] Obtain an initial feature extractor, where the initial feature extractor includes a feature matching network, and the feature matching network includes a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer;

[0008] Obtain a first training sample set, where the first training sample set includes multiple first anomaly samples;

[0009] According to the first training sample set, perform a first training on the initial feature extractor to obtain a first feature extractor;

[0010] Obtain a second training sample set, where the second training sample set includes multiple second anomaly samples and anomaly labels corresponding to the second anomaly samples;

[0011] According to the second training sample set, perform a second training on the first feature extractor to optimize the parameters of the feature matching network and obtain a trained feature extractor.

[0012] Optionally, the feature matching network is used to perform feature extraction processing and feature matching processing on the image to be detected input to the feature matching network.

[0013] Optionally, the implementation formula of the feature matching network is:

[0014]

[0015]

[0016]

[0017] Wherein, represents the similarity matrix, and respectively represent the features of a test image and the features in the memory; is the first pooling layer, is the second pooling layer, is the output of the first pooling layer, is the output of the second pooling layer.

[0018] Optionally, according to the first training sample set, the initial feature extractor is first trained to obtain the first feature extractor, including:

[0019] According to the first training sample set, the initial feature extractor is first trained. During the first training process, the first loss function is obtained, and the gradient descent algorithm is used to minimize the first loss function to optimize the parameters of the first feature extractor, and the trained first feature extractor is obtained;

[0020] Wherein, the formula of the first loss function is:

[0021]

[0022]

[0023] Wherein, is the first loss function, is the first reference score, which is used to encourage the model to focus on the low scores of difficult samples or abnormal samples, is the second reference score, represents the abnormal score of the i-th image during the first training process, is the number of image samples, H is the size of the feature map in the vertical direction, W is the size of the feature map in the horizontal direction, and r is the sensitivity in adjusting the feature matching process.

[0024] Optionally, according to the second training sample set, the first feature extractor is second trained to optimize the parameters of the feature matching network, and the trained feature extractor is obtained, including:

[0025] According to the second training sample set, the first feature extractor is secondarily trained. During the second training process, a second loss function is obtained, and the gradient descent algorithm is used to minimize the second loss function to optimize the parameters of the feature matching network, thereby obtaining a trained feature extractor.

[0026] Optionally, obtaining the second training sample set includes:

[0027] Obtain multiple normal samples;

[0028] Using the data augmentation method CutPaste, based on multiple normal samples, create a second training sample set.

[0029] Optionally, after secondarily training the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtaining a trained feature extractor, it further includes:

[0030] Obtain the image to be detected;

[0031] Input the image to be detected into the trained feature extractor to obtain the abnormal data corresponding to the image to be detected through the trained feature extractor.

[0032] In a second aspect, an anomaly detection device based on differentiable feature matching provided by the present application includes:

[0033] A first acquisition module, configured to acquire an initial feature extractor, where the initial feature extractor includes a feature matching network, and the feature matching network includes a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer;

[0034] A second acquisition module, configured to acquire a first training sample set, where the first training sample set includes multiple first abnormal samples;

[0035] A first training module, configured to primarily train the initial feature extractor according to the first training sample set to obtain a first feature extractor;

[0036] A third acquisition module, configured to acquire a second training sample set, where the second training sample set includes multiple second abnormal samples and abnormal labels corresponding to the second abnormal samples;

[0037] A second training module, configured to secondarily train the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtain a trained feature extractor.

[0038] Optionally, the first acquisition module is further configured to:

[0039] The feature matching network is configured to perform feature extraction processing and feature matching processing on the image to be detected input to the feature matching network.

[0040] Optionally, the first acquisition module is further configured to:

[0041] A feature matching network, and the implementation formula of the feature matching network is:

[0042]

[0043]

[0044]

[0045] Wherein, represents a similarity matrix, and respectively represent the features of a test image and the features in the memory; is the first pooling layer, is the second pooling layer, is the output of the first pooling layer, is the output of the second pooling layer.

[0046] Optionally, the first training module is further configured to:

[0047] According to the first training sample set, perform first training on the initial feature extractor. During the first training process, obtain the first loss function, and use the gradient descent algorithm to minimize the first loss function to optimize the parameters of the first feature extractor, and obtain the trained first feature extractor;

[0048] Wherein, the formula of the first loss function is:

[0049]

[0050]

[0051] Wherein, is the first loss function, is the first reference score, which is used to encourage the model to focus on the low scores of difficult samples or abnormal samples, is the second reference score, represents the abnormal score of the i-th image during the first training process, is the number of image samples, H is the size of the feature map in the vertical direction, W is the size of the feature map in the horizontal direction, and r is to adjust the sensitivity in the feature matching process.

[0052] Optionally, the second training module is further configured to:

[0053] According to the second training sample set, the first feature extractor is second-trained. During the second training process, a second loss function is obtained, and the gradient descent algorithm is used to minimize the second loss function to optimize the parameters of the feature matching network, thereby obtaining a trained feature extractor.

[0054] Optionally, the third acquisition module is further configured to:

[0055] Acquire a plurality of normal samples;

[0056] Using the data augmentation method CutPaste, based on the plurality of normal samples, a second training sample set is created.

[0057] Optionally, the second training module is further configured to:

[0058] Acquire an image to be detected;

[0059] Input the image to be detected into the trained feature extractor to obtain abnormal data corresponding to the image to be detected through the trained feature extractor.

[0060] In a third aspect, the present application provides an anomaly detection device based on differentiable feature matching, including:

[0061] A processor and a memory;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory, causing the processor to execute various possible implementation manners in the first aspect as described above.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement various possible implementation manners in the first aspect.

[0065] In a fifth aspect, the present application provides a computer program product, which when executed by a processor is used to implement various possible implementation manners in the first aspect.

[0066] An anomaly detection method, device, and equipment based on differentiable feature matching provided by this application obtain an initial feature extractor, where the initial feature extractor includes a feature matching network; obtain a first training sample set; perform a first training on the initial feature extractor according to the first training sample set to obtain a first feature extractor; obtain a second training sample set; perform a second training on the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtain a trained feature extractor. Thus, by using the feature extractor and its corresponding feature matching network, the discrete matching algorithm is converted into a differentiable module, and the joint optimization of the feature extractor and the learnable feature matching module is achieved through training. In addition, after performing the first training on the initial feature extractor, continue to perform the second training on the first feature extractor according to the second training sample set, and adopt a two-stage interactive training method to ensure the accuracy and reliability of anomaly detection, solve the technical problem of low efficiency of anomaly detection based on differentiable feature matching, and achieve the technical effect of improving the efficiency of anomaly detection based on differentiable feature matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0068] Figure 1 It is a schematic diagram of the solution of the anomaly detection method provided by an embodiment of this application;

[0069] Figure 2 It is the flow of the anomaly detection method based on differentiable feature matching provided by an embodiment of this application Figure 1 ;

[0070] Figure 3 It is the flow of the anomaly detection method based on differentiable feature matching provided by an embodiment of this application Figure 2 ;

[0071] Figure 4 It is a schematic diagram of the architecture of the differentiable feature matching framework provided by an embodiment of this application;

[0072] Figure 5 It is a schematic diagram of the structure of the anomaly detection device based on differentiable feature matching provided by an embodiment of this application;

[0073] Figure 6 It is a hardware structure diagram of the anomaly detection equipment based on differentiable feature matching provided by an embodiment of this application.

[0074] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by reference to specific embodiments. Detailed Description of the Embodiments

[0075] Here, exemplary embodiments will be described in detail, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0076] In the prior art, the anomaly detection method based on differentiable feature matching in the prior art overly relies on a pre-trained backbone model, has low detection accuracy and poor detection efficiency, and there is a technical problem of low anomaly detection effectiveness.

[0077] In view of the above problems, an anomaly detection method, device and equipment based on differentiable feature matching provided by the present application convert a discrete matching algorithm into a differentiable module by using a feature extractor and its corresponding feature matching network, and realize the joint optimization of the feature extractor and the learnable feature matching module through training. In addition, after the initial feature extractor is subjected to the first training, the first feature extractor is continuously subjected to the second training according to the second training sample set, and a two-stage interactive training method is adopted to ensure the accuracy and reliability of anomaly detection, solve the technical problem of low anomaly detection effectiveness based on differentiable feature matching, and achieve the technical effect of improving anomaly detection effectiveness.

[0078] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0079] Figure 1 This is the schematic diagram of the solution of the anomaly detection method provided by the embodiments of the present application. As Figure 1 shown, Figure 1 (a) and Figure 1 (b) are the schematic diagrams of the solutions of the existing anomaly detection methods, Figure 1Figure (c) is the schematic diagram of the anomaly detection method provided in this application. In the figure, the red markers are used to indicate that the corresponding modules are trainable or changeable modules, the blue boxes are used to indicate multiple image features (Patch Feature) extracted, the black triangles are used to indicate sampling operations (Sampling Operation), the black arrows are used to indicate the forward process (Forward Process), the brown arrows are used to indicate the propagation path of the gradient flow (Gradient Flow), and the dashed boxes are used to indicate that the module is a learnable adapter (Learnable Adapter);

[0080] Among them, Figure 1 (a) and Figure 1 (b) show the traditional anomaly detection method based on feature matching. In this method, the feature extractor and the feature matching module are separate. The feature extractor usually uses an encoder (Encoder) to extract image features and uses a self-supervised framework (Self-Supervised Learning) to train the feature extractor through proxy tasks; while the feature matching module relies on simple nearest neighbor matching or other static algorithms (Matching Algorithm) for anomaly detection; a major limitation of this method is that the feature extractor and the feature matching module are optimized separately, resulting in a probability of incompatibility between the feature extractor and the feature matching module, thus affecting the performance of anomaly detection;

[0081] Figure 1 (c) proposes a differentiable feature matching framework that integrates feature extraction and feature matching into a differentiable feature matching model (Differentiable Feature Matching Module). In this model, the features extracted by the feature extractor are directly passed to the feature matching network, realizing learnable feature matching. This method allows the joint optimization of the feature extractor and the feature matching network through gradient descent, thus improving the performance and efficiency of anomaly detection.

[0082] Figure 2 The flow of the anomaly detection method based on differentiable feature matching provided in the embodiments of this application Figure 1 . As Figure 2 shown, the anomaly detection method based on differentiable feature matching provided in the embodiments of this application includes:

[0083] S201. Obtain an initial feature extractor, where the initial feature extractor includes a feature matching network;

[0084] In this embodiment, the feature matching network includes a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer. The feature matching network is used to perform feature extraction processing and feature matching processing on the image to be detected input to the feature matching network. The implementation formula of the feature matching network is:

[0085]

[0086]

[0087]

[0088] Among them, represents the similarity matrix, and respectively represent the features of a test image and the features in the memory; is the first pooling layer, is the second pooling layer, is the output of the first pooling layer, is the output of the second pooling layer.

[0089] Obtain an initial feature extractor that integrates the feature matching network;

[0090] It should be noted that the initial feature extractor is not only responsible for basic feature extraction tasks, but also embeds a feature matching network to enhance the anomaly detection ability; the feature matching network has a complex and delicate structure, including a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer; these hierarchical designs enable the FMN to perform in-depth feature extraction processing and accurate feature matching processing on the image to be detected input to the network, so as to more accurately identify the abnormal regions in the image.

[0091] S202. Obtain the first training sample set;

[0092] In this embodiment, the first training sample set includes multiple first abnormal samples.

[0093] Obtain a sample set containing multiple first abnormal samples and use this sample set as the first training sample set.

[0094] S203. Perform the first training on the initial feature extractor according to the first training sample set to obtain the first feature extractor;

[0095] Use the first training sample set to perform initial training on the initial feature extractor so that it can learn basic abnormal features. After the training is completed, the corresponding first feature extractor is obtained.

[0096] S204. Obtain the second training sample set;

[0097] In this embodiment, the second training sample set includes multiple second abnormal samples and abnormal labels corresponding to the second abnormal samples.

[0098] Obtain a sample set containing multiple second abnormal samples and abnormal labels corresponding to the second abnormal samples, and use this sample set as the second training sample set.

[0099] S205. According to the second training sample set, perform second training on the first feature extractor to optimize the parameters of the feature matching network and obtain a trained feature extractor.

[0100] Use the second training set to further train the first feature extractor to optimize the parameters of the feature matching network. After the training is completed, obtain a trained feature extractor.

[0101] An anomaly detection method, device, and equipment based on differentiable feature matching provided by an embodiment of this application. By obtaining an initial feature extractor, where the initial feature extractor includes a feature matching network; obtaining a first training sample set; according to the first training sample set, performing first training on the initial feature extractor to obtain a first feature extractor; obtaining a second training sample set; according to the second training sample set, performing second training on the first feature extractor to optimize the parameters of the feature matching network and obtain a trained feature extractor. Thus, by adopting the feature extractor and its corresponding feature matching network, the discrete matching algorithm is converted into a differentiable module, and through training, the joint optimization of the feature extractor and the learnable feature matching module is achieved. In addition, after performing first training on the initial feature extractor, continue to perform second training on the first feature extractor according to the second training sample set, adopting a two-stage interactive training method to ensure the accuracy and reliability of anomaly detection, solve the technical problem of low anomaly detection efficiency, and achieve the technical effect of improving anomaly detection efficiency.

[0102] Figure 3 It is the flow of the anomaly detection method based on differentiable feature matching provided by an embodiment of this application. Figure 2 As Figure 3 shown, on the basis of the above embodiment, this embodiment makes a supplementary description of the process of obtaining the feature extractor, including:

[0103] S301. Obtain a first training sample set. According to the first training sample set, perform first training on the initial feature extractor. During the first training process, obtain a first loss function.

[0104] In this embodiment, the formula of the first loss function is:

[0105]

[0106]

[0107] Among them, is the first loss function, is the first reference score, which is used to encourage the model to focus on the low scores of difficult samples or abnormal samples. is the second reference score, represents the anomaly score of the i-th image in the first training process. is the number of image samples, H is the size of the feature map in the vertical direction, W is the size of the feature map in the horizontal direction, and r is the sensitivity in adjusting the feature matching process.

[0108] It should be noted that the first reference score is an image-level reference score, and the second reference score is a pixel-level reference score. The reference score dimension of the second reference score is higher than that of the first reference score.

[0109] Obtain the first training sample set. According to the first training sample set, perform the first training on the initial feature extractor so that it can initially learn to distinguish the features of normal and abnormal samples. During the first training process, obtain the first loss function, which measures the gap between the current prediction ability of the model and the ideal state.

[0110] S302. Use the gradient descent algorithm to minimize the first loss function to optimize the parameters of the first feature extractor and obtain the trained first feature extractor.

[0111] Use the gradient descent algorithm to minimize the first loss function, gradually adjust the parameters of the first feature extractor to reduce the prediction error, and optimize its feature extraction ability for normal and abnormal samples. Finally, obtain the trained first feature extractor.

[0112] S303. Obtain multiple normal samples and use the data augmentation method CutPaste to create a second training sample set based on the multiple normal samples.

[0113] Obtain multiple normal samples and use the data augmentation method CutPaste. By adding or replacing local abnormal regions to the normal samples and creating a second training sample set that includes multiple second abnormal samples and the corresponding abnormal labels based on the multiple normal samples.

[0114] S304. According to the second training sample set, perform the second training on the first feature extractor. During the second training process, obtain the second loss function.

[0115] In this embodiment, the formula of the second loss function is the same as that of the first loss function, and will not be elaborated in this embodiment.

[0116] According to the second training sample set, the initially trained first feature extractor is secondarily trained. During the second training process, in combination with the output of the feature matching network, a second loss function is further obtained. This function comprehensively considers the synergistic effect between the feature extractor and the feature matching network, as well as the accuracy of the model for anomaly detection.

[0117] S305. Use the gradient descent algorithm to minimize the second loss function to optimize the parameters of the feature matching network, and obtain the trained feature extractor.

[0118] Use the gradient descent algorithm to minimize the second loss function to optimize the parameters of the feature extractor and the feature matching network. Since the feature extractor and the feature matching network are integrated into an end-to-end model, through gradient backpropagation, the parameters of both can be adjusted simultaneously to make them work better in coordination, and obtain the trained feature extractor and the optimized feature matching network.

[0119] S306. Obtain the image to be detected, and input the image to be detected into the trained feature extractor to obtain the anomaly data corresponding to the image to be detected through the trained feature extractor.

[0120] Obtain the image to be detected, input the image to be detected into the trained feature extractor, extract the image features through the feature extractor, and perform matching and anomaly detection on the image features by the feature matching network in the feature extractor to obtain the anomaly data corresponding to the image to be detected.

[0121] An anomaly detection method based on differentiable feature matching provided by an embodiment of the present application. By obtaining a first training sample set, the initial feature extractor is first trained according to the first training sample set. During the first training process, a first loss function is obtained, and the gradient descent algorithm is used to minimize the first loss function to optimize the parameters of the first feature extractor, obtaining a trained first feature extractor; multiple normal samples are obtained, and the data augmentation method CutPaste is used to create a second training sample set based on the multiple normal samples. According to the second training sample set, the first feature extractor is second trained. During the second training process, a second loss function is obtained; the gradient descent algorithm is used to minimize the second loss function to optimize the parameters of the feature matching network, obtaining a trained feature extractor, and an image to be detected is obtained. The image to be detected is input into the trained feature extractor to obtain anomaly data corresponding to the image to be detected through the trained feature extractor; thus, by using the first training sample set and the second training sample set, the diversity of the training samples is increased, and the recognition ability of abnormal features is improved. At the same time, through the first loss function and the second loss function, not only the gap between the current prediction ability of the model and the ideal state is measured, but also the synergistic effect between the feature extractor and the feature matching network is comprehensively considered, ensuring the accuracy of anomaly detection. In addition, the gradient descent algorithm is used to further improve the robustness, accuracy, and rapidity of the model in feature extraction and feature matching, solving the technical problem of low anomaly detection efficiency and achieving the technical effect of improving anomaly detection efficiency.

[0122] Figure 4 It is a schematic diagram of the architecture of the differentiable feature matching framework provided by an embodiment of the present application, as Figure 4 shown, the model architecture of the differentiable feature matching framework provided by the present application mainly includes an input image, a feature extractor, a feature matching network (FMN), and an output image;

[0123] The trapezoidal module in the figure is the feature extractor, which is used to extract image features from the input image and initialize the parameters (Parameter Initialization) of these image features. These features are used for subsequent feature matching. Among them, the feature extractor can be any pre-trained neural network model, such as a convolutional neural network (CNN), etc.;

[0124] The FMN includes a matching layer, multiple adapter layers, and multiple pooling layers. Among them, the multiple adapter layers include a first adapter layer and a second adapter layer, and the pooling layers include a first pooling layer and a second pooling layer. The first pooling layer is a minimum pooling layer (Min Pool), and the second pooling layer is a maximum pooling layer (Max Pool);

[0125] The matching layer is used to receive the features from the feature extractor and calculate their similarity with the features in the memory; the features in the memory are sampled from normal images and are used to provide a comparison benchmark for the test image features; The first adapter layer: located after the matching layer, is used to modulate the matching results in order to extract richer information. The adapter layer can be a linear layer (Conv), a convolutional layer (MLP), or an attention layer (Attention), etc.; The first pooling layer: performs a pooling operation on the matching results modulated by the adapter layer to reduce the computational amount and extract key information; The second adapter layer: further modulates the pooled results to extract more useful feature information; The second pooling layer: performs pooling again on the output of the second adapter layer to obtain the final matching result;

[0126] Since FMN integrates feature extraction and feature matching into one model, the joint optimization of the feature extractor and FMN can be achieved through gradient descent. Thus, the corresponding anomaly detection image and the corresponding anomaly part (Anomaly Score) are output for the input image.

[0127] A differentiable feature matching framework provided by an embodiment of the present application integrates feature extraction and feature matching into an end-to-end model and introduces a learnable feature matching network, realizing the joint optimization of the feature extractor and the feature matching network, thereby improving the accuracy, speed, and adaptability of anomaly detection, solving the technical problem of low efficiency of anomaly detection, and achieving the technical effect of improving the efficacy of anomaly detection.

[0128] Figure 5 It is a schematic structural diagram of an anomaly detection device based on differentiable feature matching provided by an embodiment of the present application. The device in this embodiment can be in the form of software and / or hardware. As Figure 5 shown, an anomaly detection device 500 based on differentiable feature matching provided by an embodiment of the present application includes: a first acquisition module 501, a second acquisition module 502, a first training module 503, a third acquisition module 504, and a second training module 505:

[0129] The first acquisition module 501 is used to acquire an initial feature extractor, where the initial feature extractor includes a feature matching network, and the feature matching network includes a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer;

[0130] The second acquisition module 502 is used to acquire a first training sample set, where the first training sample set includes a plurality of first anomaly samples;

[0131] The first training module 503 is configured to perform a first training on an initial feature extractor according to a first training sample set to obtain a first feature extractor;

[0132] The third acquisition module 504 is configured to acquire a second training sample set, where the second training sample set includes a plurality of second abnormal samples and abnormal labels corresponding to the second abnormal samples;

[0133] The second training module 505 is configured to perform a second training on the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtain a trained feature extractor.

[0134] In a possible implementation, the first acquisition module 501 is further configured to:

[0135] The feature matching network is configured to perform feature extraction processing and feature matching processing on a to-be-detected image input to the feature matching network.

[0136] In a possible implementation, the first acquisition module 501 is further configured to:

[0137] The feature matching network, and the implementation formula of the feature matching network is:

[0138]

[0139]

[0140]

[0141] Wherein, represents a similarity matrix, and respectively represent the features of a test image and the features in the memory; is the first pooling layer, is the second pooling layer, is the output of the first pooling layer, is the output of the second pooling layer.

[0142] In a possible implementation, the first training module 503 is further configured to:

[0143] Perform a first training on the initial feature extractor according to the first training sample set. During the first training, obtain a first loss function and use the gradient descent algorithm to minimize the first loss function to optimize the parameters of the first feature extractor and obtain a trained first feature extractor;

[0144] Wherein, the formula of the first loss function is:

[0145]

[0146]

[0147] Among them, is the first loss function, is the first reference score, which is used to encourage the model to focus on the low scores of difficult samples or abnormal samples, is the second reference score, represents the anomaly score of the i-th image in the first training process, is the number of image samples, H is the size of the feature map in the vertical direction, W is the size of the feature map in the horizontal direction, and r is the sensitivity in adjusting the feature matching process.

[0148] In a possible implementation, the second training module 505 is further configured to:

[0149] According to the second training sample set, perform second training on the first feature extractor. During the second training process, obtain the second loss function, and use the gradient descent algorithm to minimize the second loss function to optimize the parameters of the feature matching network and obtain the trained feature extractor.

[0150] In a possible implementation, the third acquisition module 504 is further configured to:

[0151] Obtain multiple normal samples;

[0152] Use the data augmentation method CutPaste to create a second training sample set based on the multiple normal samples.

[0153] In a possible implementation, the second training module 505 is further configured to:

[0154] Obtain the image to be detected;

[0155] Input the image to be detected into the trained feature extractor to obtain the anomaly data corresponding to the image to be detected through the trained feature extractor.

[0156] An anomaly detection device based on differentiable feature matching provided by an embodiment of the present application can implement the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0157] Figure 6 This is the hardware structure diagram of the anomaly detection based on differentiable feature matching provided by an embodiment of the present application. As Figure 6 shown, the anomaly detection device 600 based on differentiable feature matching includes:

[0158] A processor 601 and a memory 602;

[0159] The memory stores computer execution instructions;

[0160] The processor executes the computer-executable instructions stored in the memory 602, so that the anomaly detection device based on differentiable feature matching executes the anomaly detection method based on differentiable feature matching as described above.

[0161] It should be understood that the above-mentioned processor 601 may be a central processing unit (abbreviation: CPU in English: Central Processing Unit), or may also be other general-purpose processors, digital signal processors (abbreviation: DSP in English: Digital Signal Processor), application specific integrated circuits (abbreviation: ASIC in English: Application Specific Integrated Circuit), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being completed by the execution of the hardware processor, or can be completed by the combination of the hardware and software modules in the processor.

[0162] The memory 602 may include high-speed random access memory (abbreviation: RAM in English: Random Access Memory), and may also include non-volatile memory (abbreviation: NVM in English: Non-volatile memory), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disc, etc.

[0163] The embodiment of the present application correspondingly provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by the processor, they are used to implement the anomaly detection method based on differentiable feature matching as described above.

[0164] The embodiment of the present application correspondingly further provides a computer program product, and when the computer program is executed by the processor, it is used to implement the anomaly detection method based on differentiable feature matching as described above.

[0165] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps may be adopted in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0166] It should be further noted that although the steps in the flowchart are shown sequentially in the direction of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear description in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential either, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0167] It should be understood that the above-described device embodiments are merely illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0168] In addition, without special instructions, in each embodiment of the present application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0169] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Without special instructions, the processor can be any suitable hardware processor, such as CPU, GPU, FPGA, DSP, and ASIC, etc. Without special instructions, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory RRAM (Resistive Random Access Memory), dynamic random access memory DRAM (Dynamic Random Access Memory), static random access memory SRAM (Static Random-Access Memory), enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), high-bandwidth memory HBM (High-Bandwidth Memory), hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0170] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs.

[0171] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0172] Those skilled in the art will readily conceive of other embodiments of this application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include the common general knowledge or conventional technical means in this technical field that are not disclosed in this application. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of this application are pointed out by the following claims.

[0173] It should be understood that this application is not limited to the precise structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is only limited by the appended claims.

Claims

1. An anomaly detection method based on differentiable feature matching, characterized in that Including: Obtain an initial feature extractor, where the initial feature extractor includes a feature matching network, and the feature matching network includes a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer; Obtain a first training sample set, where the first training sample set includes multiple first abnormal samples; Perform a first training on the initial feature extractor according to the first training sample set to obtain a first feature extractor; Obtain a second training sample set, where the second training sample set includes multiple second abnormal samples and abnormal labels corresponding to the second abnormal samples; Perform a second training on the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtain a trained feature extractor.

2. The method according to claim 1, wherein The feature matching network is used to perform feature extraction processing and feature matching processing on the image to be detected input to the feature matching network.

3. The method according to claim 2, wherein The implementation formula of the feature matching network is: Among them, represents a similarity matrix, and respectively represent the features of a test image and the features in the memory; is the first pooling layer, is the second pooling layer, is the output of the first pooling layer, is the output of the second pooling layer.

4. The method according to any one of claims 1 to 3, characterized in that, The step of performing a first training on the initial feature extractor according to the first training sample set to obtain a first feature extractor includes: Perform a first training on the initial feature extractor according to the first training sample set. During the first training, obtain a first loss function and use the gradient descent algorithm to minimize the first loss function to optimize the parameters of the first feature extractor and obtain the trained first feature extractor; Wherein, the formula of the first loss function is: Among them, is the first loss function, is the first reference score, which is used to encourage the model to focus on the low scores of difficult samples or abnormal samples. is the second reference score, represents the anomaly score of the i-th image in the first training process, is the number of image samples, H is the size of the feature map in the vertical direction, W is the size of the feature map in the horizontal direction, and r is the sensitivity in adjusting the feature matching process.

5. The method according to any one of claims 1 to 3, characterized in that, The step of performing a second training on the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtain a trained feature extractor includes: Perform a second training on the first feature extractor according to the second training sample set. During the second training, obtain a second loss function and use the gradient descent algorithm to minimize the second loss function to optimize the parameters of the feature matching network and obtain the trained feature extractor.

6. The method according to claim 5, wherein The step of obtaining the second training sample set includes: Obtain multiple normal samples; Use the data augmentation method CutPaste to create a second training sample set based on the multiple normal samples.

7. The method according to any one of claims 1 to 3, characterized in that After performing a second training on the first feature extractor according to the second training sample set to optimize the parameters of the feature matching network and obtain a trained feature extractor, it further includes: Obtain an image to be detected; Input the image to be detected into the trained feature extractor to obtain abnormal data corresponding to the image to be detected through the trained feature extractor.

8. An anomaly detection device based on differentiable feature matching, characterized in that, Including: A first acquisition module, configured to obtain an initial feature extractor, where the initial feature extractor includes a feature matching network, and the feature matching network includes a matching layer, a first adapter layer, a first pooling layer, a second adapter layer, and a second pooling layer; A second acquisition module, configured to obtain a first training sample set, where the first training sample set includes multiple first abnormal samples; A first training module, configured to perform first training on the initial feature extractor according to the first training sample set to obtain a first feature extractor; A third acquisition module, configured to acquire a second training sample set, where the second training sample set includes a plurality of second abnormal samples and abnormal labels corresponding to the second abnormal samples; A second training module, configured to perform second training on the first feature extractor according to the second training sample set to optimize parameters of the feature matching network and obtain a trained feature extractor.

9. An anomaly detection device based on differentiable feature matching, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the abnormal detection method based on differentiable feature matching according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the abnormal detection method based on differentiable feature matching according to any one of claims 1 to 7.

Images