Anomaly detection method and device, equipment, storage medium

Abstract

The application provides an anomaly detection method, which comprises: acquiring a to-be-detected image; and determining a detection result of the to-be-detected image according to the to-be-detected image and an anomaly detection model, wherein the anomaly detection model is obtained by training according to a first loss function comprising a sample weight parameter, and the sample weight parameter represents the importance of normal samples or abnormal samples for model training during training. The anomaly detection method of the embodiment of the application effectively solves the problem caused by unbalanced training sample data by using the anomaly detection model obtained by training according to the first loss function comprising the sample weight parameter to predict the anomaly of the to-be-detected image, so that the detection result is more accurate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an anomaly detection method, apparatus, device, and storage medium. Background Technology

[0002] Detecting anomalies is crucial in many fields, including industrial manufacturing, routine system inspections, and surveillance video analysis. For example, in railway freight, ensuring safe train operation requires collecting numerous images of trains, which are then manually analyzed to detect anomalies. In power transmission line inspection, drones and other equipment are used to collect images of the lines, which are then manually inspected. In industrial manufacturing, surface quality inspection before products leave the factory ensures product quality and reduces customer complaints.

[0003] However, manual anomaly detection is labor-intensive, increasing costs for businesses and departments. Furthermore, the monotonous nature of the work can lead to worker fatigue, resulting in missed or false positives. In recent years, with the rise of deep learning, artificial intelligence (AI) technology has been widely applied in daily life and production. Using AI to replace manual anomaly detection saves manpower, reducing costs for businesses and departments; moreover, machine algorithms are fatigue-free, leading to more consistent detection results.

[0004] In actual production practice, the number of normal and abnormal samples collected is often unbalanced (often with more normal samples than abnormal samples), resulting in imbalanced training data for machine learning algorithm models. Anomaly detection models trained using existing machine learning algorithms often suffer from unstable training processes, extremely high training difficulty, and unsatisfactory anomaly detection performance due to this imbalanced training data.

[0005] Application content

[0006] The embodiments of this application provide an anomaly detection method and apparatus, which solve the problem that the anomaly detection model is unstable during training due to imbalanced training sample data, making model training extremely difficult and resulting in the anomaly detection performance of the trained model failing to meet requirements.

[0007] In a first aspect, this application provides an anomaly detection method, comprising first acquiring an image to be detected; and then determining a detection result of the image to be detected based on the image to be detected and an anomaly detection model, wherein the anomaly detection model is trained according to a first loss function including sample weight parameters, the sample weight parameters representing the importance of normal or abnormal samples used for training to the model training.

[0008] The anomaly detection method of this application uses an anomaly detection model trained with a first loss function including sample weight parameters to predict anomalies in the image to be detected, effectively solving the problem caused by imbalanced training sample data and obtaining more accurate detection results.

[0009] In one possible implementation, to facilitate users in reviewing the model detection results in subsequent steps, the method further includes: generating a detection result distribution map based on the above detection results, wherein the detection result distribution map includes the distribution of images with detection results of normal, abnormal, or questionable; and presenting the detection result distribution map to the user.

[0010] In one possible implementation, the detection result distribution map further includes a confidence region and a doubtful region; images distributed within the doubtful region are provided to the user for manual verification, preventing false detections by the detection model and further ensuring the accuracy of the detection results.

[0011] In another possible implementation, considering the improvement in the accuracy of the anomaly detection model's detection results during subsequent training and optimization processes, as well as the workload of manual verification, the range of the suspicious area in the detection result distribution map is set to be adjustable based on the user's instructions. That is, the user can reduce the range of the suspicious area according to the actual situation during use to reduce the workload of manual verification, or increase the range of the suspicious area to increase the accuracy of the detection results.

[0012] In another possible implementation, the sample weight parameters are determined based on the clustering analysis results of the normal sample set and the abnormal sample set in the training sample data. That is, the anomaly detection model can automatically adjust dynamically according to the increase / decrease of normal or abnormal samples in the training sample data without the involvement of algorithm experts, thereby improving the update efficiency of the anomaly detection model.

[0013] In another possible implementation, the anomaly detection model is also trained based on a second loss function, which constrains the structure of the feature space of the anomaly detection model, thereby shortening the distance between features of normal samples and increasing the distance between features of anomaly samples and features of normal samples in the feature space, increasing the accuracy of the detection results output by the anomaly detection model, and facilitating the generation of a distribution map of the detection results.

[0014] In another possible implementation, the training sample data of the anomaly detection model includes anomaly samples obtained through augmentation. The augmentation method includes: first, acquiring multiple normal samples and at least one anomaly sample; then comparing the normal samples and the anomaly sample, and determining the anomaly region of the anomaly sample based on the comparison result; then extracting features from the normal samples and the anomaly sample respectively to obtain a first feature map representing the normal samples and a second feature map representing the anomaly samples; replacing the features of the region corresponding to the anomaly region in the first feature map with the features of the anomaly region in the second feature map to obtain a first fused feature map; and obtaining augmented anomaly samples based on the first fused feature map and the image generation model to enrich the types of anomaly sample data and increase the number of anomaly samples to balance the number of anomaly samples with the number of normal samples.

[0015] In another possible implementation, the above augmentation method may further include: acquiring multiple normal samples, at least one abnormal sample, and at least one abnormal feature; comparing the normal samples and the abnormal samples, and determining a replaceable region in the normal samples based on the comparison result; determining a replacement region in the replaceable region based on a user instruction; extracting features from the normal samples to obtain a third feature map characterizing the normal samples; replacing the features of the replacement region in the normal samples with the abnormal features to obtain a second fused feature map; and obtaining the augmented abnormal sample based on the second fused feature map and an image generation model.

[0016] In another possible implementation, the above anomaly detection method further includes: selecting images whose detection results are abnormal according to the manual verification results and / or the output of the anomaly detection model as anomaly samples for optimized training, using the selected anomaly samples to optimize the training of the anomaly detection model, and continuously improving the detection accuracy of the anomaly detection model and expanding the number of anomaly sample types that the anomaly detection model can cover.

[0017] Secondly, this application also provides an anomaly detection device, comprising:

[0018] The acquisition module is used to acquire the image to be detected;

[0019] The detection module is used to determine the detection result of the image to be detected based on the image to be detected and the anomaly detection model, wherein the anomaly detection model is trained according to a first loss function including sample weight parameters, and the sample weight parameters represent the importance of normal or abnormal samples used for training to the model training.

[0020] In another possible implementation, the device further includes:

[0021] A visualization module is used to generate a detection result distribution map based on the detection results, wherein the detection result distribution map includes the distribution of images with detection results of normal, abnormal, or questionable.

[0022] The distribution map of the detection results is presented to the user.

[0023] In another possible implementation, the detection result distribution map also includes a confidence region and a doubtful region;

[0024] The visualization module is also used to provide users with images distributed within the suspected area, so that the users can manually verify the suspected images.

[0025] In another possible implementation, the device further includes:

[0026] The adjustment module is used to adjust the range of the suspicious area in the detection result distribution map based on the user's instructions.

[0027] In another possible implementation, the sample weight parameters are determined based on the clustering analysis results of the normal sample set and the abnormal sample set in the training sample data.

[0028] In another possible implementation, the anomaly detection model is also trained based on a second loss function, which constrains the structure of the feature space of the anomaly detection model, such that the distance between features of normal samples is shortened and the distance between features of anomaly samples and features of normal samples is increased in the feature space.

[0029] In another possible implementation, the training sample data of the anomaly detection model includes anomalous samples obtained through augmentation, the augmentation method comprising:

[0030] Obtain multiple normal samples and at least one abnormal sample;

[0031] By comparing the normal sample and the abnormal sample, the abnormal region of the abnormal sample is determined based on the comparison result;

[0032] The features of the normal sample and the features of the abnormal sample are extracted respectively to obtain a first feature map representing the normal sample and a second feature map representing the abnormal sample;

[0033] The features of the region corresponding to the abnormal region in the first feature map are replaced with the features of the abnormal region in the second feature map to obtain the first fused feature map.

[0034] Based on the first fused feature map and the image generation model, augmented abnormal samples are obtained.

[0035] In another possible implementation, the training sample data of the anomaly detection model includes anomalous samples obtained through augmentation, the augmentation method comprising:

[0036] Acquire multiple normal samples, at least one abnormal sample, and at least one abnormal feature;

[0037] By comparing the normal sample and the abnormal sample, the replaceable region in the normal sample is determined based on the comparison result;

[0038] The replacement area is determined based on user instructions within the replaceable area;

[0039] Features of normal samples are extracted to obtain a third feature map representing normal samples;

[0040] The features of the replacement region in the normal sample are replaced with the abnormal features to obtain the second fused feature map;

[0041] Based on the second fused feature map and the image generation model, augmented anomalous samples are obtained.

[0042] In another possible implementation, the device further includes:

[0043] An optimization module is used to select images whose manual verification results are abnormal as abnormal samples for optimization training, and to optimize and train the anomaly detection model using the selected abnormal samples.

[0044] Thirdly, this application also provides an anomaly detection device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in the first aspect or any possible implementation of the first aspect.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0046] Fifthly, this application also provides a computer program or computer program product, characterized in that the computer program or computer program product includes instructions that, when executed, implement the method described in the first aspect or any possible implementation of the first aspect.

[0047] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0048] Figure 1The architecture diagram and training and inference process diagrams of the anomaly detection device provided in the embodiments of this application are shown.

[0049] Figure 2 This is a schematic diagram of the architecture of the abnormal sample augmentation module provided in this embodiment;

[0050] Figure 3 A flowchart of an abnormal sample augmentation method provided in the embodiments of this application;

[0051] Figure 4 A flowchart illustrating another abnormal sample augmentation method provided in this application embodiment;

[0052] Figure 5 A schematic diagram illustrating the construction process of the anomaly detection model provided in this application embodiment;

[0053] Figure 6 A flowchart illustrating the method for constructing an anomaly detection model provided in this application embodiment;

[0054] Figure 7 This is a schematic diagram of component localization in a railway freight anomaly detection scenario provided in this application embodiment;

[0055] Figure 8 A visualization of the detection results of an image to be detected in a railway freight anomaly detection scenario provided in this application embodiment;

[0056] Figure 9 A visualization of the detection results of the image to be detected in the railway freight anomaly detection scenario provided in this application embodiment, when the detection results include suspicious results;

[0057] Figure 10 This is a schematic diagram of component detection model inference in an electronic device assembly and inspection scenario provided in this application embodiment;

[0058] Figure 11 A schematic diagram of the anomaly detection device is provided for the embodiments of this application;

[0059] Figure 12 A schematic diagram of the anomaly detection device is provided for the embodiments of this application. Detailed Implementation

[0060] This application provides an anomaly detection method and apparatus. This method and apparatus can enrich training data by generating more realistic anomaly samples using weakly supervised data generation techniques, even when the number of anomaly samples is limited. Based on this, a data-adaptive anomaly detection method is designed, which minimizes human intervention in the algorithm model training and update process, improving automation and reducing costs. In practical use, through human-machine collaboration, anomaly detection accuracy is improved while reducing human intervention, and the model is continuously iterated and updated, constantly strengthening the advantages of this method and apparatus.

[0061] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0063] It is understood that the anomaly detection method provided in this application can be applied to any processing device with image processing capabilities. This processing device can be a terminal with a Central Processing Unit (CPU) and / or a Graphics Processing Unit (GPU), wherein the terminal includes, but is not limited to, a Personal Computer (PC), a workstation, etc. The processing device can also be a server with a CPU and / or a GPU; the server can be standalone or a server cluster. In some cases, the terminal and the server can also work together to implement the above-described anomaly detection method.

[0064] Figure 1 This is an architectural diagram of an anomaly detection device provided in an embodiment of this application. Figure 1As shown, the anomaly detection device includes at least an anomaly sample augmentation module 11 and an anomaly detection module 13. The anomaly sample augmentation module 11 is used to obtain a large number of augmented anomaly samples based on a large number of normal samples and a small number of anomaly samples collected, enriching the types of anomaly samples and initially balancing the number of anomaly samples and normal samples. The large number of normal samples, the small number of anomaly samples, and the large number of augmented anomaly samples constitute the training sample data. The training sample data is used to train a deep learning network model to obtain a qualified anomaly detection model. The anomaly detection module 13 is used to detect whether the input image to be detected is abnormal, and feeds back images with abnormal detection results to the training sample data to train and update the anomaly detection model, thereby continuously improving the detection accuracy of the anomaly detection model and expanding the number of anomaly sample types that the anomaly detection model can cover.

[0065] In some embodiments, the anomaly detection device further includes a manual screening module 12. The manual screening module 12 is used to determine available anomaly samples from a large number of augmented anomaly samples according to user instructions, so as to prevent low-quality samples in the augmented anomaly samples from affecting the training effect.

[0066] It is understood that normal samples are images of the object to be inspected in a normal state obtained by actual photography (e.g., a fault-free train or a properly assembled workpiece), while abnormal samples are images of the object to be inspected in an abnormal state obtained by actual photography (e.g., a faulty train or an incorrectly assembled workpiece). Usable abnormal samples are images with high image clarity and rich variations in the shape and color of abnormal areas, selected by industry experts or ordinary people from a large number of augmented abnormal samples based on prior knowledge. For ease of description, normal samples, abnormal samples, and augmented abnormal samples will be referred to as normal sample images, abnormal sample images, and augmented abnormal sample images, respectively.

[0067] In other embodiments, the anomaly detection device further includes an interactive sample visualization module 14 and a manual review module 15. The interactive sample visualization module 14 visualizes the detection results to facilitate user review. Then, the manual review module 15 outputs the final detection results and identifies missed and false-detection anomaly sample images based on received instructions, feeding them back to the anomaly sample augmentation module 11. The anomaly sample augmentation module continues to generate more augmented anomaly sample images based on the feedback anomaly sample images and normal sample images. The manual screening module 12 then filters out usable anomaly sample images from the augmented anomaly sample images. The selected usable anomaly sample images are added to the training sample data to continue training and updating the anomaly detection model. This process is repeated iteratively to continuously improve the detection accuracy of the anomaly detection model and expand the number of anomaly sample types that the anomaly detection model can cover.

[0068] The following section introduces the working principle and specific structure of the abnormal sample augmentation module.

[0069] Figure 2 This is a schematic diagram of the architecture of the anomalous sample augmentation module. (Example:) Figure 2 As shown, the abnormal sample augmentation module includes at least an image feature extraction unit 111, an image feature fusion unit 112, an image generation unit 113, and an image authenticity discrimination unit 114.

[0070] The image sample augmentation module first identifies the abnormal regions in the abnormal sample images. Then, the image feature extraction unit 111 extracts features from both the normal sample images and the abnormal sample images. The extracted features are then sent to the image feature fusion unit 112. The image feature fusion unit 112 replaces the image features of the corresponding regions in the normal sample images with the image features of the abnormal regions in the abnormal sample images to obtain the fused image features. The fused image features are then input into the image generation unit 113. The image generation unit 113 uses the image generation model therein to output the generated image of the abnormal sample.

[0071] Next, we will combine the appendix Figure 3 The workflow of the abnormal sample augmentation module is explained in detail.

[0072] Figure 3 A flowchart of an abnormal sample augmentation method provided in an embodiment of this application.

[0073] In step S301, the abnormal sample augmentation module acquires multiple normal sample images and at least one abnormal sample image. It is easy to understand that normal sample images are images obtained by a camera terminal (e.g., a video camera) capturing the object to be detected in a normal state, while abnormal sample images are images obtained by a camera terminal capturing the object to be detected in an abnormal state. The abnormal sample augmentation module can acquire normal and abnormal sample images from the capturing terminal, or from a terminal that stores normal and abnormal sample images.

[0074] It should be explained that the multiple normal sample images and abnormal sample images here all include images of the same object to be detected, but the shooting angle, shooting environment, etc. of the multiple normal sample images may be different from those of the abnormal sample images.

[0075] Step S302 determines the abnormal regions of the abnormal sample images.

[0076] Specifically, this involves comparing abnormal sample images with normal sample images and determining the abnormal regions of the abnormal samples based on the comparison results. That is, identifying the regions where the pixel values ​​or image features differ between the abnormal sample images and the normal sample images. There are various comparison methods. For example, registration algorithms (such as SIFT+RANSAC, ECC, etc.) can be used to register normal and abnormal sample images. Then, corresponding regions in the normal and abnormal sample images are compared to obtain the regions with differences in pixel values ​​or image features; these regions with differences are the abnormal regions.

[0077] Step S303: Obtain the first feature map representing the normal sample image and the second feature map representing the abnormal sample image.

[0078] The image feature extraction unit 111 extracts features from normal sample images and abnormal sample images respectively to obtain a first feature map representing the normal sample image and a second feature map representing the abnormal sample image. The image feature extraction unit 111 can extract features from normal sample images and abnormal sample images in any manner. For example, it can use a feature extraction layer of a convolutional neural network to extract features from both normal and abnormal sample images to obtain the first and second feature maps.

[0079] Furthermore, it should be explained that the first and second feature maps here are not limited to specific feature maps. For example, they can be color feature maps, texture feature maps, intermediate layer feature maps extracted by deep learning models, or edge feature maps of images.

[0080] Step S304: Determine the first fused feature map based on the first feature map, the second feature map, and the abnormal region.

[0081] The image feature fusion unit 112, based on the first feature map representing the normal sample image, the second feature map representing the abnormal sample image, and the abnormal region obtained in the above steps, replaces the features of the region corresponding to the abnormal region in the first feature map with the features of the abnormal region in the second feature map to obtain the first fused feature map.

[0082] It should be explained that in this step, the image feature fusion unit 112 can directly replace the features of the abnormal region in the first feature map with the features of the abnormal region in the second feature map to obtain the first fused feature map; or it can process the features of the abnormal region in the second feature map (e.g., edge extraction, brightness adjustment, edge smoothing, distortion, etc.) and then replace the features of the abnormal region in the first feature map with the features of the abnormal region in the first feature map to obtain the first fused feature map.

[0083] Finally, the image feature fusion unit 112 executes step S305 to input the first fused feature map into the image generation model to obtain the image of the augmented abnormal sample, thereby realizing the augmentation of the abnormal sample data.

[0084] The image generation model here can be obtained by training a deep learning model. For example, this deep learning model can be a variational autoencoder (VAE), and the training sample pairs are feature maps representing real images and real images, such as normal sample images and / or abnormal sample images from the training sample data. Specifically, feature maps of normal sample images and / or abnormal sample images are extracted and obtained. The feature maps representing normal sample images and normal sample images, and / or the feature maps representing abnormal sample images and abnormal sample images constitute the training sample pairs to train the variational autoencoder and obtain the image generation model.

[0085] The feature maps representing real images from the training sample pairs are input into the variational autoencoder (VAC). The VAC predicts the real images represented by the input feature maps. A loss is calculated based on the predicted real images and the real images from the training sample pairs. The VAC parameters are then updated based on this loss, thus completing model training. Training stops when the loss meets the termination condition, such as when the loss converges. The trained VAC can then be used as an image generation model to output images of anomalous samples based on the fused input feature maps.

[0086] For example, this deep learning model can also be a GAN (Generative Adversarial Network). A GAN consists of two parts: an image generator and an image realism discriminator. The ideal effect of a GAN is achieved through the game and adversarial interaction between the image generator and the image realism discriminator. For instance, in the early stages of training, the first-generation image generator can generate some poor-quality anomalous sample images based on the fused feature maps of the input. The first-generation image realism discriminator can accurately classify these augmented anomalous sample images. In short, the image realism discriminator is a binary classifier, outputting 0 for augmented anomalous sample images and 1 for real images. Then, the second-generation image generator is trained. The second-generation image generator can generate slightly better anomalous sample images, allowing the first-generation image realism discriminator to recognize these augmented anomalous sample images as real images. Finally, a second-generation image realism discriminator is trained, which can accurately distinguish between real images and anomalous sample images generated by the second-generation image generator. Following this pattern, there will be third, fourth, and many more generations of image generators and image realism discriminators. Finally, the image realism discriminator will be unable to distinguish between augmented anomalous sample images and real images. At this point, the GAN has been fitted, and the image generator can output augmented anomalous sample images that meet the requirements based on the fused feature maps.

[0087] The abnormal sample augmentation module uses the trained image generation model to obtain a large number of augmented abnormal sample images. These large number of augmented abnormal sample images are input into the manual screening module 12. Based on the user's selection instructions, the module selects abnormal sample images with high image clarity, rich variations in the shape and color of the abnormal area, and characteristics such as the location and shape of the abnormal area that conform to the prior knowledge of relevant industry experts and ordinary people. These images are then added to the training sample data to continue training and updating the abnormal detection model.

[0088] Understandably, in order to facilitate the use of augmented anomalous sample images for training anomaly detection models, image generation models automatically add labels (i.e., labels indicating that they are anomalous sample images) to the output augmented anomalous sample images, reducing the workload of manual annotation.

[0089] In another embodiment, another method for augmenting anomalous samples is also provided. This is discussed below in conjunction with... Figure 4 This augmentation method is described in detail.

[0090] like Figure 4 As shown, the augmentation method includes the following steps:

[0091] S401. Acquire multiple normal sample images, at least one abnormal sample image, and at least one abnormal feature.

[0092] The method for obtaining multiple normal sample images and at least one abnormal sample image is similar to the method in step S301 described above, and can be found in step S301; it will not be repeated here. It is understandable that abnormal features are features that may appear abnormal on the object to be detected. For example, if the object to be detected is a train, the abnormality that may appear on the train is a crack, then the abnormal feature is the image feature of the crack image.

[0093] S402. Determine replaceable regions based on normal and abnormal sample images.

[0094] Register normal and abnormal sample images to determine the image region of the object to be detected in the normal sample image. This region is then identified as a replaceable region to avoid adding abnormal features to the background region, which would render the augmented abnormal sample image invalid.

[0095] Then, the replacement region is determined within the replaceable region based on user instructions via S403. Unlike the previous augmentation method, this method uses a manually specified approach, allowing for more flexible addition of anomalous features and resulting in richer and more diverse augmented anomalous sample images.

[0096] The implementation methods of obtaining the third feature map representing the normal sample image, replacing the features of the replacement region of the third feature map with abnormal features to obtain the second fused feature map and the image generation model in steps S404 to S406 are similar to the implementation methods of obtaining the first feature map representing the normal sample image, replacing the features of the region corresponding to the abnormal region of the first feature map and the image generation model in the previous augmentation method. Please refer to the above text, and it will not be repeated here.

[0097] The following is in conjunction with the appendix Figure 5 This section details the construction methods for anomaly detection models.

[0098] like Figure 5 As shown, the anomaly detection model is trained using at least the training sample data and a first loss function including sample weight parameters. This trained anomaly detection model can address the problem of an imbalance between the number of normal and abnormal sample images in the training sample data, resulting in more accurate detection results.

[0099] The following is in conjunction with the appendix Figure 6 This section details the construction process of the anomaly detection model.

[0100] First, step S601 acquires multiple normal sample data and multiple abnormal sample data. It should be noted that the multiple abnormal sample data include abnormal sample images augmented by the augmentation module and abnormal sample images acquired during data collection. Both the normal and abnormal sample data are labeled sample images.

[0101] Then, in step S602, cluster analysis is performed on multiple normal sample data and multiple abnormal sample data respectively to obtain cluster analysis results. Specifically, image features of normal sample data and abnormal sample data are first extracted, and then cluster analysis is performed on the extracted image features to obtain cluster analysis results. In this embodiment of the application, any clustering method can be used to perform cluster analysis on the image features of normal samples and the image features of abnormal samples, and this application is not limited thereto. For example, it can be a traditional clustering method such as K-means + automatic category selection, spectral clustering, or any new clustering method.

[0102] Step S603: Determine the sample weight parameters of the first loss function based on the cluster analysis results.

[0103] Cluster analysis results include the number of clusters for normal samples and the number of samples in each normal sample cluster, as well as the number of clusters for abnormal samples and the number of samples in each abnormal sample cluster. See the following formula for details:

[0104] w p =C n / (n p (C p +C n ))

[0105] w n =C p / (n n (C p +C n ))

[0106] In the formula: w p The weight of a specific normal sample; w n The weight of a specific outlier sample; C p and C n These represent the number of clusters for normal samples and abnormal samples, respectively, i.e., the number of variation patterns in the samples; n p n represents the number of samples in the cluster containing a specific normal sample. n This indicates the number of samples in the cluster containing a specific abnormal sample.

[0107] In this way, the more samples there are and the fewer the variation patterns of the current sample (n p or n n The larger the value of the sample (C), the lower its weight; the more varied the change patterns of the other type of sample (C... n Or C p To achieve a balancing effect, the weight of this type should also be higher, so as to avoid the overwhelming sample of this type by the excessive variety of other types of variation patterns.

[0108] For a specific sample (i.e., a sample of a certain object to be detected), the loss function is defined as:

[0109] L = -w n ·y·log(p)-w p (1-y)log(1-p)

[0110] Where w p and w n y is the sample weight calculated in S603, y is the sample label (1 represents an abnormal sample, 0 represents a normal sample), p is the probability of the sample being abnormal predicted by the model, and log is the logarithmic function.

[0111] The first loss function is obtained by summing the loss functions of the above-mentioned specific samples; the first loss function is the corresponding... Figure 5 The adaptive weight classification loss function in [the context].

[0112] Finally, in step S604, an anomaly detection model is obtained by training a network model based on multiple normal sample data, multiple abnormal sample data, and a first loss function. This network model can be a convolutional neural network (CNN), meaning the extraction of sample image features during the clustering analysis can be achieved through the convolutional layers of the CNN model. Specifically, normal and abnormal sample data are input into the CNN model in batches, and the feature extraction layer of the CNN model extracts the image features of both normal and abnormal samples.

[0113] The above-mentioned method for constructing an anomaly detection model allows the model to automatically and dynamically adjust sample weight parameters based on the increase or decrease of normal or abnormal samples in the training sample data. This reduces the degree of human intervention in the initial and ongoing model updates during training, lowers the involvement of algorithm experts during use, and reduces maintenance costs.

[0114] In another embodiment, the anomaly detection model is also based on a second loss function (i.e., corresponding to...). Figure 5 The auxiliary recognition loss function is trained and obtained. Based on the idea of ​​metric learning, the auxiliary loss function applies structural constraints to the feature space of the convolutional neural network. By requiring that the features of normal samples are close to each other and the features of abnormal samples are far away from the features of normal samples, the distribution of normal and abnormal samples in the learned feature space meets certain structural constraints. This enables the subsequent interactive sample visualization module 14 to efficiently and accurately distinguish between normal and abnormal samples when visualizing the samples.

[0115] Specifically, for a pair of samples, the loss is defined as:

[0116]

[0117] Where x i and x jy represents the features of the i-th and j-th images in the training data; i and y j The label represents the image, as defined in the previous section; α is an adjustable hyperparameter; ||·|| represents the norm (modulus) of the vector.

[0118] The trained anomaly detection model can determine whether the input image to be detected is abnormal. The anomaly detection model outputs the judgment result to the interactive sample distribution visualization module 14. Since the second loss function is used to explicitly constrain the feature space structure during the training process of the anomaly detection model, the distribution of normal samples and abnormal samples has a certain regularity. Therefore, it can be visualized to help staff quickly distinguish between normal samples and abnormal samples, thereby quickly completing the review.

[0119] By performing dimensionality reduction processing on the features of normal and abnormal samples during the training of the anomaly detection model (dimensionality reduction methods can include PCA, t-SNE, autoencoders, etc.), two-dimensional feature points are obtained. The feature point distribution is then plotted in the same coordinate system to obtain the distribution of images where the detection results are normal, abnormal, or questionable, facilitating the presentation of the detection result distribution map to the user. The detection result distribution map includes confidence regions and questionable regions. The interactive sample distribution visualization module 14 can provide the user with sample images where the feature points of the sample to be detected fall into the questionable region, allowing the user to manually verify the questionable images.

[0120] The above detection result distribution map can be obtained based on the distribution of feature points of normal samples and feature points of abnormal samples during statistical training. For example, it can obtain the predicted classification boxes for segmenting the feature points of normal samples and feature points of abnormal samples, so that the feature points of normal samples are distributed within the predicted classification box area, and the feature points of abnormal samples are distributed outside the predicted classification box area.

[0121] To prevent false positives from the anomaly detection model, the predicted classification box can be scaled down proportionally while maintaining its center, resulting in a confidence box. The area selected by the confidence box is the confidence region, and the area between the confidence box and the predicted classification box is the doubtful region. If a feature point in the image to be detected falls into the confidence region, the detection result is directly output as normal. If a feature point in the image to be detected falls outside the predicted classification box, the detection result is directly output as an anomaly. If a feature point in the image to be detected falls into the doubtful region, the image is presented to the user for verification to confirm whether it is an anomaly. Therefore, the user only needs to review the image corresponding to the feature points located in the doubtful region, thereby reducing the workload of review and improving review efficiency.

[0122] The manual review module 15 determines whether the suspected images to be detected are abnormal according to the user's instructions, and finally outputs the detection results. The images that are found to be abnormal by the review results are used as abnormal samples for optimization training to optimize the anomaly detection model.

[0123] Specifically, the anomaly detection device feeds back the verified results of the abnormal sample images to the anomaly sample augmentation module, which then continues to augment the received abnormal sample images to obtain more abnormal sample images, thereby optimizing and updating the anomaly detection model.

[0124] In another embodiment, the anomaly detection device also uses images that are detected as abnormal by the anomaly detection model as abnormal samples for optimization training, and performs optimization training on the anomaly detection model.

[0125] In one embodiment, the above-mentioned anomaly detection device can be applied to anomaly detection scenarios in railway freight transportation.

[0126] Specifically, the process involves acquiring normal and abnormal sample images of freight trains via camera terminals, then analyzing the corresponding anomaly types. For example, one type of anomaly is an anomaly on a specific component (such as a broken spring or a leaking bearing), while another type is an anomaly with an unfixed location (such as a foreign object). Different annotation strategies are adopted to annotate the images based on the anomaly type. For instance, if the image is an anomaly on a specific component, the location of the component in the image needs to be annotated (e.g., the location of the normal component and the location of the abnormal component). If the image is an anomaly with an unfixed location, the location where the anomaly occurs needs to be annotated.

[0127] Then, different detection schemes are used for detection based on different anomaly types. For anomaly types on specific components, a component detection model is first trained to locate the components in the image (see...). Figure 7 Based on this, the aforementioned anomaly detection device is used to detect each component, wherein the anomaly sample augmentation module augments the number of anomaly sample images for each component. Then, the anomaly detection module trains an anomaly detection model for each component to determine its abnormal state.

[0128] For anomalies with non-fixed locations, the above-mentioned anomaly detection device can be used directly for detection. The anomaly sample augmentation module augments the number of anomaly sample images until the number of anomaly sample images is equal to the number of normal sample images. Then, the anomaly detection model is trained using the anomaly sample images and normal sample images to determine the anomaly state of the image to be detected.

[0129] Both of the above-mentioned anomaly detection schemes can display the detection results by outputting them (see...). Figure 8 ).

[0130] The interactive sample distribution visualization module can filter out images of samples that require manual review (see...). Figure 9 The highlighted image is the image of the sample that needs to be manually reviewed. After the staff reviews the image of the sample that needs to be reviewed, the images of the sample that was identified incorrectly (missed detection, false detection) are fed back to the abnormal sample augmentation module. Then, the abnormal sample augmentation module continues to generate new abnormal sample images based on the feedback images. The new abnormal sample images are added to the training sample data to continue training the abnormal detection model, so as to improve the performance of the abnormal detection model.

[0131] In another embodiment, the above-described anomaly detection device can be applied to electronic equipment assembly inspection scenarios.

[0132] Specifically, the process involves acquiring images of the electronic device assembly process via a camera terminal, then labeling the correctly assembled components in these images (e.g., using bounding boxes). The labeled images are then used as training samples to train a component detection model. If the detection boxes of the component detection model coincide with the labeled boxes, the corresponding component is considered normal and added to the normal sample library; otherwise, the corresponding component is considered abnormal and added to the abnormal sample library (see [link to documentation]). Figure 10 ).

[0133] Then, the aforementioned anomaly detection device is used to detect sample images in the normal sample library, thereby filtering out samples that are falsely detected by the model.

[0134] Images detected as abnormal by the anomaly detection device can be manually reviewed. The number of normal components detected in the image is counted and compared with the number of normal components that should appear in the image to determine if an anomaly exists. The abnormal images are then fed back to the anomaly sample augmentation module, which generates new anomaly sample images based on the feedback images. These new anomaly sample images are added to the training sample data to further train the anomaly detection model, thereby improving its performance.

[0135] This application also provides an anomaly detection device 2, Figure 11 This is a schematic diagram of the anomaly detection device 2, as shown below. Figure 11 As shown, the device includes at least:

[0136] Acquisition module 21 is used to acquire the image to be detected;

[0137] The detection module 22 is used to determine the detection result of the image to be detected based on the image to be detected and the anomaly detection model, wherein the anomaly detection model is trained according to a first loss function including sample weight parameters, and the sample weight parameters represent the importance of normal or abnormal samples used for training to the model training.

[0138] In another possible implementation, the device further includes:

[0139] The visualization module 23 is used to generate a detection result distribution map based on the detection results, wherein the detection result distribution map includes the distribution of images with detection results of normal, abnormal or questionable;

[0140] The distribution map of the detection results is presented to the user.

[0141] In another possible implementation, the detection result distribution map also includes a confidence region and a doubtful region;

[0142] The visualization module 23 is also used to provide the user with images distributed in the area of ​​suspicion, so that the user can manually verify the images of suspicion.

[0143] In another possible implementation, the device further includes:

[0144] The adjustment module 24 is used to adjust the range of the suspicious area in the detection result distribution map based on the user's instructions.

[0145] In another possible implementation, the sample weight parameters are determined based on the clustering analysis results of the normal sample set and the abnormal sample set in the training sample data.

[0146] In another possible implementation, the anomaly detection model is also trained based on a second loss function, which constrains the structure of the feature space of the anomaly detection model, such that the distance between features of normal samples is shortened and the distance between features of anomaly samples and features of normal samples is increased in the feature space.

[0147] In another possible implementation, the training sample data of the anomaly detection model includes anomalous samples obtained through augmentation, the augmentation method comprising:

[0148] Obtain multiple normal samples and at least one abnormal sample;

[0149] By comparing the normal sample and the abnormal sample, the abnormal region of the abnormal sample is determined based on the comparison result;

[0150] The features of the normal sample and the features of the abnormal sample are extracted respectively to obtain a first feature map representing the normal sample and a second feature map representing the abnormal sample;

[0151] The features of the region corresponding to the abnormal region in the first feature map are replaced with the features of the abnormal region in the second feature map to obtain the first fused feature map.

[0152] Based on the first fused feature map and the image generation model, augmented abnormal samples are obtained.

[0153] In another possible implementation, the training sample data of the anomaly detection model includes anomalous samples obtained through augmentation, the augmentation method comprising:

[0154] Acquire multiple normal samples, at least one abnormal sample, and at least one abnormal feature;

[0155] By comparing the normal sample and the abnormal sample, the replaceable region in the normal sample is determined based on the comparison result;

[0156] The replacement area is determined based on user instructions within the replaceable area;

[0157] Features of normal samples are extracted to obtain a third feature map representing normal samples;

[0158] The features of the replacement region in the normal sample are replaced with the abnormal features to obtain the second fused feature map;

[0159] Based on the second fused feature map and the image generation model, augmented anomalous samples are obtained.

[0160] In another possible implementation, the device further includes:

[0161] The optimization module 25 is used to select images whose manual verification results are abnormal as abnormal samples for optimization training, and to use the selected abnormal samples to optimize and train the anomaly detection model.

[0162] The anomaly detection device 2 according to the embodiments of this application can correspond to the execution of the method described in the embodiments of this application, and the above and other operations and / or functions of each module in the anomaly detection device 2 are respectively for implementing Figure 3 , Figure 4 , Figure 6 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0163] It should also be noted that the embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0164] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the methods described above.

[0165] This application also provides a computer program or computer program product that includes instructions that, when executed, cause a computer to perform any of the methods described above.

[0166] Figure 12 A schematic diagram of the anomaly detection device provided in this application.

[0167] As shown in Figure 12, the anomaly detection device 100 includes a processor 101, a memory 102, a communication interface 103, and a bus 104. The processor 101, memory 102, and communication interface 103 communicate via the bus 104, or via wireless transmission or other means. The memory 102 stores executable program code, and the processor 101 can call the program code stored in the memory 102 to execute the anomaly detection method in the aforementioned method embodiments.

[0168] It should be understood that, in the embodiments of this application, the processor 101 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0169] The memory 102 may include read-only memory and random access memory, and provides instructions and data to the processor 101. The memory 102 may also include non-volatile random access memory. For example, the memory 102 may also store a training dataset.

[0170] The memory 102 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0171] In addition to the data bus, bus 104 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 104 in the diagram.

[0172] It should be understood that the anomaly detection device 100 according to the embodiments of this application may correspond to the anomaly detection device in the embodiments of this application, and may correspond to the execution of the anomaly detection device according to the embodiments of this application. Figure 3 , Figure 4 , Figure 6 The corresponding entities in the method shown, and the above and other operations and / or functions of each device in the anomaly detection device 100, are respectively implemented to achieve Figure 3 , Figure 4 , Figure 6 The corresponding processes of each method will not be elaborated here for the sake of brevity.

[0173] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0174] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, processor-executed software modules, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0175] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An anomaly detection method characterized by, The method comprises: acquiring an image to be detected; determining a detection result of the image to be detected according to the image to be detected and an anomaly detection model, wherein the anomaly detection model is trained according to a first loss function comprising a sample weight parameter, and the sample weight parameter represents the importance of normal samples or abnormal samples for model training; the sample weight parameter comprises a weight parameter of a normal sample and a weight parameter of an abnormal sample, the weight parameter of the normal sample is determined based on the number of clusters of the normal sample, the number of clusters of the abnormal sample and the number of samples in the cluster where the normal sample is located, wherein the size of the weight parameter of the normal sample is positively correlated with the number of clusters of the abnormal sample and negatively correlated with the number of samples in the cluster where the normal sample is located, the weight parameter of the abnormal sample is determined based on the number of clusters of the abnormal sample, the number of clusters of the normal sample and the number of samples in the cluster where the abnormal sample is located, wherein the size of the weight parameter of the abnormal sample is positively correlated with the number of clusters of the normal sample and negatively correlated with the number of samples in the cluster where the abnormal sample is located, and the number of clusters of the normal sample, the number of samples in the cluster where the normal sample is located, the number of clusters of the abnormal sample and the number of samples in the cluster where the abnormal sample is located are obtained based on clustering analysis of training samples in a training sample set.

2. The method of claim 1, wherein, The method further comprises generating a detection result distribution map according to the detection result, wherein the detection result distribution map comprises the distribution of images with normal, abnormal or suspicious detection results; presenting the detection result distribution map to a user.

3. The method of claim 2, wherein, The detection result distribution map further comprises a confidence region and a suspicious region; providing images distributed in the suspicious region to the user for manual verification of the suspicious images by the user.

4. The method of claim 3, wherein, The method further comprises: adjusting the range of the suspicious region in the detection result distribution map based on the instructions of the user.

5. The method of claim 1, wherein, The anomaly detection model is further trained based on a second loss function, and the second loss function is used to constrain the structure of a feature space of the anomaly detection model, so that the distance between features of normal samples is shortened and the distance between features of abnormal samples and features of normal samples is increased in the feature space.

6. The method of claim 1, wherein, The training sample data of the anomaly detection model comprises abnormal samples obtained through augmentation, and the augmentation method comprises: acquiring a plurality of normal samples and at least one abnormal sample; comparing the normal samples and the abnormal sample, and determining an abnormal region of the abnormal sample based on the comparison result; extracting features of the normal samples and features of the abnormal sample respectively to obtain a first feature map representing the normal samples and a second feature map representing the abnormal sample; replacing the features of the abnormal region corresponding region in the first feature map with the features of the abnormal region in the second feature map to obtain a first fusion feature map; generating an augmented abnormal sample based on the first fusion feature map and an image generation model.

7. The method of claim 1, wherein, The training sample data of the anomaly detection model comprises abnormal samples obtained through augmentation, and the augmentation method comprises: obtaining a plurality of normal samples, at least one abnormal sample and at least one abnormal feature; comparing the normal samples and the abnormal samples, and determining a replaceable region in the normal samples based on a comparison result; determining a replacement region in the replaceable region based on a user instruction; extracting a feature of the normal sample to obtain a third feature map representing the normal sample; replacing a feature of the replacement region in the normal sample with the abnormal feature to obtain a second fusion feature map; generating an augmented abnormal sample based on the second fusion feature map and an image generation model.

8. The method of claim 3 or 4, wherein, The method further includes: selecting an image with an abnormal result of the manual verification as an abnormal sample for optimization training, and optimizing the abnormal detection model by using the selected abnormal sample.

9. An abnormality detection device characterized by comprising: comprise: an acquisition module, configured to acquire a to-be-detected image; a detection module, configured to determine a detection result of the to-be-detected image according to the to-be-detected image and an abnormal detection model, wherein the abnormal detection model is trained according to a first loss function comprising a sample weight parameter, and the sample weight parameter represents importance of a normal sample or an abnormal sample for model training during training. The sample weight parameter comprises a weight parameter of a normal sample and a weight parameter of an abnormal sample. The weight parameter of the normal sample is determined based on a cluster quantity of the normal sample, a cluster quantity of the abnormal sample and a sample quantity of a cluster where the normal sample is located. The weight parameter of the normal sample is positively correlated with the cluster quantity of the abnormal sample and is negatively correlated with the sample quantity of the cluster where the normal sample is located. The weight parameter of the abnormal sample is determined based on the cluster quantity of the abnormal sample, the cluster quantity of the normal sample and the sample quantity of the cluster where the abnormal sample is located. The weight parameter of the abnormal sample is positively correlated with the cluster quantity of the normal sample and is negatively correlated with the sample quantity of the cluster where the abnormal sample is located. The cluster quantity of the normal sample, the sample quantity of the cluster where the normal sample is located, the cluster quantity of the abnormal sample and the sample quantity of the cluster where the abnormal sample is located are obtained based on cluster analysis on training samples in a training sample set.

10. The apparatus of claim 9, wherein, The device further comprises: a visualization module, configured to generate a detection result distribution map according to the detection result, wherein the detection result distribution map comprises distribution of images with a normal detection result, an abnormal detection result or a doubtful detection result; presenting the detection result distribution map to a user.

11. The apparatus of claim 10, wherein, The detection result distribution map further comprises a confidence region and a doubtful region. The visualization module is further configured to provide images distributed in the doubtful region to the user, so that the user manually verifies the doubtful images.

12. The apparatus of claim 11, wherein, The device further comprises: an adjustment module, configured to adjust a range of the doubtful region in the detection result distribution map based on an instruction of the user.

13. The apparatus of claim 9, wherein, The abnormal detection model is further trained based on a second loss function. The second loss function is used to constrain a structure of a feature space of the abnormal detection model, so that distances between features of normal samples are shortened and distances between features of abnormal samples and features of normal samples are lengthened in the feature space.

14. The apparatus of claim 9, wherein, The training sample data of the anomaly detection model includes an abnormal sample obtained through augmentation, and the augmentation method includes: Obtaining a plurality of normal samples and at least one abnormal sample; Comparing the normal samples and the abnormal sample, and determining an abnormal area of the abnormal sample based on a comparison result; Respectively extracting features of the normal samples and the abnormal sample to obtain a first feature map representing the normal samples and a second feature map representing the abnormal sample; Replacing features of a corresponding area of the abnormal area in the first feature map with features of the abnormal area in the second feature map to obtain a first fusion feature map; Based on the first fusion feature map and the image generation model, an augmented abnormal sample is obtained.

15. The apparatus of claim 9, wherein, The training sample data of the anomaly detection model includes an abnormal sample obtained through augmentation, and the augmentation method includes: Obtaining a plurality of normal samples, at least one abnormal sample and at least one abnormal feature; Comparing the normal samples and the abnormal sample, and determining a replaceable area in the normal sample based on a comparison result; Determining a replacement area in the replaceable area based on a user instruction; Extracting features of the normal sample to obtain a third feature map representing the normal sample; Replacing features of the replacement area in the normal sample with the abnormal features to obtain a second fusion feature map; Based on the second fusion feature map and the image generation model, an augmented abnormal sample is obtained.

16. The apparatus of claim 11 or 12, wherein, The device further includes: An optimization module for selecting an image with an abnormal result of the manual verification as an abnormal sample for optimization training, and optimizing the training of the anomaly detection model using the selected abnormal sample.

17. An anomaly detection device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-8.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed in the computer, the computer executes the method of any one of claims 1-8.

19. A computer program product, characterised in that, The computer program product includes instructions that, when executed, implement the method of any one of claims 1-8.

Images