Image anomaly detection method and device, equipment and storage medium

Abstract

The invention relates to the field of image processing, financial science and technology and medical health, and discloses an image anomaly detection method, which comprises the following steps: receiving a request for carrying out anomaly detection on a to-be-detected image, extracting global features of the to-be-detected image, extracting regional features of the to-be-detected image, and fusing the global features and the regional features to obtain fused features; calculating the similarity between the fusion feature and each sample feature, and selecting the maximum similarity value in each known category from a similarity result; constructing a distance measurement function according to each maximum similarity value, and calculating an overall deviation value of the regional features and the distribution features of each clustering center in spatial distribution by using the distance measurement function; and comparing the overall deviation value with a preset threshold value, judging the category to which the initial region belongs according to a comparison result, and marking a category label. According to the method, the global and local features are integrated, the similarity and the deviation value are quantified, and a reasonable threshold value is set to carry out classification decision, so that safer and more reliable image or data analysis is realized.

Description

Technical Field

[0001] The present invention relates to the fields of image processing, financial technology and medical health, and in particular to an image anomaly detection method, device, equipment and storage medium. Background Art

[0002] In the field of image and video analysis, especially in key application scenarios such as autonomous driving, security monitoring, finance, and healthcare, accurately identifying Out-of-Distribution (OOD) objects - samples that have never been seen by the detection model or are difficult to correctly classify - is crucial to ensuring the safety and reliability of the system. With the development of deep learning technology, although methods such as ODIN (outlier detection through input modification) and Mahalanobis distance have improved in some aspects, these traditional methods still face the following problems when processing complex images.

[0003] 1. Limitations of a single feature extractor: In the fields of financial technology and health care, traditional OOD anomaly detection methods often rely on a single feature extractor, which is incapable of capturing the relationship between financial key information areas or lesion areas and the global background. For example, in lung CT scans, different types of tumors may have similar appearance characteristics, but their pathological states are completely different. When relying solely on a fixed feature extractor to distinguish between benign nodules and malignant tumors, misjudgments may occur due to the inability to fully understand the relationship between local lesions and the overall anatomical structure.

[0004] 2. Lack of explicit learning mechanism for unknown financial fraud types or unknown lesion types: Existing OOD anomaly detection methods are usually trained on known categories and lack sufficient adaptability to newly emerged or unseen financial fraud types or lesion types. For example, if a doctor encounters a new disease that has never been seen before and inputs its X-ray into an existing automatic diagnosis system, the system is likely to generate overconfident but wrong prediction results. This is because traditional models do not have a learning mechanism for unknown categories, which causes them to tend to classify any object that is different from the training data set into the closest known category, thereby increasing the false alarm rate. For example, in an autonomous driving scenario, a newly introduced large wild animal that has never appeared in the area suddenly appears on the road, such as a new deer species with a strange appearance and large body. This deer has unique stripes and colors, which are significantly different in appearance from common animals (such as common deer, cattle, sheep, etc.) included in the training data set of the autonomous driving system, causing collision accidents. When the camera of an autonomous vehicle captures the image of this new type of deer, if the image is input into the existing animal recognition system based on traditional technology, the system is likely to make an incorrect judgment and mistakenly classify this new type of deer as a cattle, greatly increasing the potential risk of insurance claims accidents.

[0005] Therefore, how to improve the accuracy of image OOD anomaly detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above, it is necessary to provide an image anomaly detection method, the purpose of which is to effectively combine the regional features, global features and uncertainty prediction of the image area, so as to improve the accuracy of the detection model in image OOD anomaly detection in complex scenarios.

[0007] In a first aspect, a method for detecting anomalies in an image is provided, the method comprising:

[0008] Receiving a request for abnormality detection on an image to be detected, extracting global features of the image to be detected using a global image encoder of a detection model, extracting regional features of an initial region of the image to be detected using a local image encoder of the detection model, and fusing the global features and the regional features to obtain fused features;

[0009] Obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set;

[0010] Constructing a distance measurement function according to each maximum similarity value, calculating the cluster center of all sample features in each known category on a preset spatial distribution, and using the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center on the spatial distribution;

[0011] The overall deviation value is compared with a preset threshold value, and the category to which the initial area belongs is determined as an unknown category or a known category according to the comparison result, and a corresponding category label is marked to respond to the request for anomaly detection.

[0012] In a second aspect, an image anomaly detection device is provided, comprising:

[0013] A fusion module is used to receive a request for abnormality detection on an image to be detected, extract global features of the image to be detected using a global image encoder of a detection model, extract regional features of an initial region of the image to be detected using a local image encoder of the detection model, and fuse the global features and the regional features to obtain fused features;

[0014] A selection module is used to obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set;

[0015] A calculation module is used to construct a distance measurement function according to each maximum similarity value, calculate the cluster center of all sample features in each known category on a preset spatial distribution, and use the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center on the spatial distribution;

[0016] The marking module is used to compare the overall deviation value with a preset threshold value, determine whether the category to which the initial area belongs is an unknown category or a known category according to the comparison result, and mark the corresponding category label to respond to the request for anomaly detection.

[0017] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned image anomaly detection method are implemented.

[0018] Compared with the prior art, the present invention extracts global features and regional features of the initial region from the image to be detected at the same time and fuses the two to understand the image content more accurately. The fused features are compared with the sample features of all known categories in the database to find the most similar category, and a preliminary assessment can be made as to whether the image to be detected belongs to any known category.

[0019] A distance measurement function for quantifying deviations is constructed based on the maximum similarity value. The distance measurement function can determine the degree of difference between the new sample and the known category, that is, whether the new sample may be an unknown category. The overall deviation in the spatial distribution between the initial regional features of the suspected anomaly and each cluster center (representing the known category) is calculated. The overall deviation value reflects the degree to which the new sample deviates from the known category distribution and is a key indicator for determining whether the sample is OOD.

[0020] Finally, the overall deviation value is compared with the preset threshold to determine whether the detected area belongs to a known category or an unknown category. If the deviation value exceeds the threshold, the area is considered to be an unknown category, indicating that there may be a new, unseen type of lesion; otherwise, it is classified as a known category.

[0021] The present invention not only improves the recognition accuracy of known categories, but also enhances the sensitivity to unknown categories or abnormal situations. It makes classification decisions by integrating global and local features, quantifying similarities and deviation values, and setting reasonable thresholds. It ultimately achieves safer and more reliable medical image analysis, which is particularly suitable for the early detection and research of complex lesions in the health care field. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of an application environment of an image anomaly detection method according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a flow chart of an image anomaly detection method provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a module of an image anomaly detection device provided by an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0026] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention.

[0027] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0030] The image anomaly detection method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can receive a request for abnormality detection of an image to be detected, extract the global features of the image to be detected by using the global image encoder of the detection model, extract the regional features of the initial area of ​​the image to be detected by using the local image encoder of the detection model, fuse the global features and the regional features to obtain the fused features; obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused features and each sample feature, obtain the similarity result set between the fused features and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set; construct a distance measurement function according to each maximum similarity value, calculate the cluster center of all sample features in each known category in spatial distribution, and use the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center in the spatial distribution; compare the overall deviation value with a preset threshold, determine whether the category to which the initial area belongs is an unknown category or a known category according to the comparison result, and mark the corresponding category label to respond to the request for abnormality detection. Not only the recognition accuracy of known categories is improved, but also the sensitivity to unknown categories or abnormal situations is enhanced. By integrating global and local features, quantifying similarities and deviation values, and setting reasonable thresholds for classification decisions, a safer and more reliable medical image analysis is finally achieved. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0031] Reference Figure 2 FIG. 1 is a flow chart of an image anomaly detection method provided by an embodiment of the present invention. The method is executed by a device.

[0032] In this embodiment, the image anomaly detection method includes:

[0033] S1. Receive a request for abnormality detection on an image to be detected, use a global image encoder of a detection model to extract global features of the image to be detected, use a local image encoder of the detection model to extract regional features of an initial area of ​​the image to be detected, and fuse the global features and the regional features to obtain fused features.

[0034] In this embodiment, an image to be detected uploaded by a user is received. For example, in the field of health care, the image to be detected is provided by a doctor or a patient to check whether there is potential lung lesions. The image to be detected is preprocessed to obtain a preprocessed image, and the preprocessing includes adjusting the image size and normalizing the pixel values.

[0035] For example, when an autonomous vehicle is driving on the road, the monitoring system of the vehicle receives an image of a new type of deer with a strange appearance and large body appearing on the road from the on-board camera. Since this type of deer has never been learned by the recognition system before, its appearance is a potential abnormality, so the monitoring system receives a request for abnormality detection on this image. The global image encoder analyzes various elements and their relationships in the entire image to obtain the global features of the image to be detected. For example, the width of the road, the distribution of surrounding trees, and the proportion of the sky in the entire picture. The local image encoder focuses on the initial area where the new type of deer is located and analyzes various detailed features in this area to obtain regional features. The global features (such as appearing near the bend, the surrounding vegetation is dense, the picture color is affected by the new type of deer, etc.) and local features (unique stripes, large body size, special body structure, etc.) are fused to obtain fused features. The pre-trained global image encoder (such as deep convolutional neural networks such as ResNet and DenseNet) is loaded, and the pre-processed image is forward propagated. The high-level semantic information of the image is gradually extracted through multi-layer convolution and pooling operations to generate global features representing the entire image to be detected.

[0036] Pre-trained target detection models (such as Faster R-CNN, YOLO, etc.) are used to automatically identify the initial areas (i.e., regions of interest) where lesions may exist in the image to be detected. These initial areas may be the locations of suspected tumors or other abnormal structures.

[0037] Load the pre-trained local image encoder, which is specially used to capture local details. Perform forward propagation on the cropped initial area to extract the feature vector that can characterize the unique properties of the area, thereby obtaining the regional features.

[0038] The projected linear layer of the detection model is used to combine the acquired global features and regional features to form a comprehensive fused feature. The fused feature can be a simple concatenation or a more complex mathematical operation, such as weighted summation or learning through additional neural network layers. The resulting fused feature contains not only the overall information of the image, but also detailed information of the specific region of interest, thereby improving the accuracy of subsequent anomaly detection.

[0039] In this embodiment, the global context information of the global feature and the regional semantic information of the regional feature are fused by the following fusion formula:

[0040]

[0041] in, Represents the fusion feature, which is used to represent the image x to be detected and its initial area The comprehensive characteristics of I P represents the projected linear layer after the encoder, which is used for feature dimensionality reduction and fusion. ⊕ represents element-by-element addition, which is used to fuse local and global features. λ is a weighting coefficient used to control the contribution ratio of local and global features. represents the local features of the initial region extracted by the local image encoder, I (g) (x) represents the global features of the entire image to be detected extracted by the global image encoder.

[0042] In one embodiment, the extracting global features of the image to be detected using a global image encoder of a detection model includes:

[0043] Adjusting the size of the image to be detected and performing normalization processing on the image to be detected to obtain a preprocessed image;

[0044] The global image encoder is used to sequentially perform forward propagation, feature map extraction and feature aggregation processing on the preprocessed image to obtain the global feature.

[0045] Resize the image to be detected to ensure that the input images have the same size so that they can be processed consistently by the detection model. Normalize the pixel values ​​of the image to be detected to make the distribution of image data more standardized, which helps to speed up the feature extraction process and improve the performance of the detection model.

[0046] Load a pre-trained global image encoder (such as ResNet, DenseNet, etc.). The global image encoder has been fully trained on large-scale datasets and can capture common visual features in the image to be detected.

[0047] The forward propagation of the preprocessed image includes: calculating the preprocessed image through the global image encoder, gradually extracting the different levels of features of the preprocessed image, and calculating the output of each layer in turn according to the network structure until the final feature representation is obtained.

[0048] Extracting the feature map includes: inputting the feature representation of the forward propagation output into the middle layer or the last convolution layer of the global image encoder, obtaining the output of the middle layer or the last convolution layer to obtain the feature map, and the feature map contains the spatial structure information of the image.

[0049] Aggregate feature processing includes: applying a pooling operation (such as global average pooling or maximum pooling) to the feature map, compressing the high-dimensional feature map of the feature map into a low-dimensional feature vector, thereby obtaining the global features of the image to be detected. This low-dimensional feature vector summarizes the information of the entire image to be detected.

[0050] In one embodiment, the extracting the regional features of the initial region of the image to be detected by using the local image encoder of the detection model includes:

[0051] Using the target detection model to identify the initial area of ​​the image to be detected;

[0052] The local image encoder is used to sequentially perform forward propagation, feature map extraction and feature aggregation processing on the initial region to obtain the regional features.

[0053] The image to be detected is preprocessed to obtain a preprocessed image. The preprocessing includes adjusting the image size and normalizing the pixel values. The preprocessed image is analyzed using a pre-trained target detection model (such as Faster R-CNN, YOLO, RetinaNet, etc.). The target detection model outputs a series of bounding boxes, each of which identifies the initial region (region of interest (ROI)) that may exist in the preprocessed image, such as the location of a suspected tumor. The initial region contains the specific details of the potential lesion.

[0054] Load a pre-trained local image encoder (which is different from the convolutional neural network of the global image encoder and can be another convolutional neural network such as ResNet, DenseNet, etc.). The local image encoder focuses on extracting detailed information in small areas.

[0055] The forward propagation of the initial region includes: calculating the initial region through a local image encoder, gradually extracting different levels of features of the initial region, and calculating the output of each layer in turn according to the network structure until the final feature representation is obtained.

[0056] Extracting the feature map includes: inputting the feature representation of the forward propagation output into the middle layer or the last convolution layer of the local image encoder, obtaining the output of the middle layer or the last convolution layer to obtain the feature map, and the feature map contains information at different positions in the initial area.

[0057] Aggregate feature processing includes: applying a pooling operation (such as global average pooling or maximum pooling) to the feature map, compressing the high-dimensional feature map of the feature map into a low-dimensional feature vector, thereby obtaining the local features of the initial region, and this low-dimensional feature vector summarizes the information of the initial region.

[0058] The training process of the global image encoder is based on a deep learning framework and a large amount of annotated data, and is obtained in the following way:

[0059] Using a deep convolutional neural network (such as ResNet, DenseNet) pre-trained on a large-scale dataset (such as ImageNet) as the initial model, according to the specific classification task of medical images (such as lung cancer detection), modify the parameters of the last layer or multiple layers of the initial model to adapt to the new classification requirements.

[0060] Collect and annotate a batch of chest X-rays or other related medical imaging data, and divide the medical imaging data into training set, validation set and test set.

[0061] Preprocess the training set, such as resizing and normalizing, and use the preprocessed medical imaging data to fine-tune the initial model. Update the weights of the last layer or some layers to adapt to the predetermined task and data distribution. Use the validation set to regularly evaluate the model performance, save the best performing model version to obtain the trained global image encoder, and confirm the overall performance of the global image encoder through the test set.

[0062] The training process of the local image encoder is the same as that of the global image encoder. The difference is that a batch of medical imaging data with lesion areas (i.e., regions of interest) in chest X-rays are collected and labeled, and the medical imaging data with regions of interest are also divided into training set, validation set, and test set.

[0063] In step S1, the problem that traditional methods rely on feature extraction at a single scale, resulting in loss of important details, is solved: for example, using only global features may not capture key details of local areas, or ignoring global information: using only local features may miss the overall structure and background information of the image. The detection model of the present invention effectively overcomes these problems by introducing a dual encoder architecture (global and local image encoders), providing a richer feature representation that includes not only the overall information of the image to be detected, but also detailed information of specific areas of interest, thereby improving the accuracy of subsequent anomaly detection.

[0064] S2. Obtain samples of all known categories for training the detection model from the database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set.

[0065] In this embodiment, the preset database is a database storing feature vectors (i.e., sample features) of all training samples for training the detection model, and each category has one or more sample features. For each known category (such as different disease types), a sample feature set of all samples under the category is obtained from the database.

[0066] For each known category, all its sample features are traversed, and the similarity between the fusion feature and each sample feature in each category is calculated using a preset similarity measurement method (such as cosine similarity, Euclidean distance, etc.). For each known category, all the calculated similarity results are sorted in descending order, and the maximum value is selected as the maximum similarity value under this category.

[0067] By selecting the maximum similarity value in each known category, it is possible to determine which known category the image to be tested is closest to. This is a basic step in classification decision making and helps identify the most likely type of disease or pathological state in the image. For example, in lung CT scan analysis, this step can help the system determine whether the lesion is more likely to be pneumonia, lung cancer, or another known type of disease.

[0068] For example, in the recognition system of autonomous vehicles, the database stores a large number of samples of known animal categories, such as common image samples of deer, cattle, sheep, dogs, etc. For each sample, the system extracts the corresponding sample features. For example, for deer samples, sample features may include common deer stripe patterns (such as plum spots), body size range, horn shape and length, ear size and shape, etc.; for cattle samples, sample features include the large and sturdy body of cattle, unique fur color distribution such as black back and white belly, and the general shape of cattle horns.

[0069] The fusion features of the new deer obtained above (a fusion of global features such as its appearance near bends, dense surrounding vegetation, and the color tone of the picture being affected by the new deer, and local features such as its unique stripes, large size, and special body structure) were respectively calculated for similarity with the sample features of each known animal category in the database, and a similarity result set between the fusion features and each known category (deer, cattle, dogs, etc.) was obtained.

[0070] In one embodiment, selecting the maximum similarity value in the corresponding known category from each similarity result set includes:

[0071] Traverse the similarity result set of each known category and sort the similarity result of each known category in descending order;

[0072] Select the maximum similarity value among each known category according to the sorting order.

[0073] For each known category (e.g., different types of lung diseases), the feature vectors (i.e., sample features) of all samples in that category are traversed. Each sample feature vector represents a feature representation of a training sample.

[0074] The similarity between the fusion feature and each sample feature vector in each category is calculated using a preset similarity measurement method (such as cosine similarity, Euclidean distance, etc.). For each known category, the similarity results between all sample feature vectors and the fusion feature are sorted from large to small, and the maximum value is selected as the maximum similarity value under the category. This maximum similarity value reflects the matching degree between the image to be detected and the most similar sample in the category.

[0075] In step S2, not only can the most likely disease type in the image to be detected be identified, but a quantitative index (i.e., the maximum similarity value) can also be provided for further classification decisions and anomaly detection, thereby improving the accuracy and reliability of detection, which is particularly important in medical image analysis.

[0076] S3. Construct a distance measurement function based on each maximum similarity value, calculate the cluster center of all sample features in each known category in the preset spatial distribution, and use the distance measurement function to calculate the overall deviation value of the distribution characteristics of the regional features and each cluster center in the spatial distribution.

[0077] In the embodiment, one of the core concepts of the present invention is to modify the fusion feature after the feature extraction is completed (i.e., calculate the overall deviation value between the regional feature and the known category in the corresponding spatial distribution feature), so as to ensure that the fusion feature can better adapt to the learning of unknown lesion types, avoid directly interfering with the original feature extraction process, and maintain the basic performance of the detection model. It allows the detection model to still make reasonable predictions when facing unknown categories, rather than simply classifying them as the closest known category.

[0078] Existing technologies usually only focus on the recognition of known categories. When encountering unknown categories, it may lead to the incorrect classification of unknown categories as the most similar known categories. Even if the prediction is wrong, the detection model may show a high confidence level, misleading doctors or other users. By introducing post-correction technology and explicit learning mechanisms, the distance measurement function is used to calculate the overall deviation value between the regional features and the distribution features of each cluster center, and the uncertainty of the initial region is accurately predicted, which effectively solves the problems existing in the existing technology and improves the recognition ability and accuracy of the detection model for unknown categories.

[0079] For each known category, a distance measurement function is used to calculate the deviation between the regional features of the image to be detected and the distribution features of the cluster center of the category, and the deviation values ​​of all categories are aggregated to obtain an overall deviation value representing the overall difference. Aggregation processing includes taking the minimum value, average value or weighted sum, etc.

[0080] In one embodiment, constructing a distance measurement function according to each maximum similarity value includes:

[0081] The maximum similarity value among each known category is stored in a preset dictionary;

[0082] The keys of the dictionary are set to be the category labels of each known category, and the values ​​of the dictionary are set to be the maximum similarity values ​​of each known category;

[0083] The set dictionary is updated to obtain the distance measurement function.

[0084] The calculated maximum similarity value for each known category is stored in a preset dictionary, and the key of the dictionary is set to the category label of each known category, and the value of the dictionary is set to the maximum similarity value of each known category. The preset dictionary is a Python dictionary object, which may be empty or contain some default values ​​in its initial state. As the algorithm is executed, this dictionary will be gradually filled according to the calculated maximum similarity value to form the final data source for distance measurement.

[0085] The dictionary associates the category label of each known category with the maximum similarity value under that category. This key-value pair relationship allows for quick search of similarity information corresponding to a specific category. By using the category label as the key, the search operation can be performed within O(1) time complexity, that is, the maximum similarity value under a category can be obtained almost instantaneously. This is particularly important when processing large-scale data sets or real-time applications.

[0086] Traverse the dictionary, take the negative value of each maximum similarity value, update the dictionary to save the converted negative value, and obtain the distance measurement function H. The distance measurement function receives the fusion features of the image to be detected and the category label of the known category as input, and returns the corresponding negative maximum similarity value as the distance.

[0087] The distance measurement function is constructed based on the converted negative value, and the formula is as follows:

[0088]

[0089] Among them, H represents the output of the distance measurement function, that is, the distance measurement between the image to be detected and all known categories. represents the operation of taking the maximum value in the range of index i from 1 to K (i.e., the number of all known categories), Sim(·,·) represents the similarity metric function, which is used to calculate the similarity between two feature vectors. Similarity metrics include cosine similarity, Euclidean distance, etc. Represents the image to be detected x and its initial area After the fusion features extracted and fused by the global and local encoders, I tIt is a process of feature extraction and fusion, t i Represents the sample features of the i-th known category.

[0090] In one embodiment, the calculating the overall deviation value of the distribution characteristics of the regional characteristics and each cluster center in the spatial distribution by using the distance measurement function includes:

[0091] Calculating the deviation value between the distribution characteristics of each cluster center and the regional characteristics by using the distance measurement function to obtain the deviation value between the initial region and each known category;

[0092] Aggregation processing is performed on all deviation values ​​to obtain the overall deviation value, wherein the aggregation processing includes taking a minimum value, an average value, or a weighted sum.

[0093] Calculate the cluster centers of all sample features in each known category on the preset spatial distribution, and find a representative cluster center in the corresponding multidimensional feature space for each known category. The cluster center can best represent the common characteristics of all samples in the category.

[0094] The cluster center can be understood as the geometric center of a group of data points (in this case, the feature vectors of image samples belonging to the same category). Specifically, it can be calculated as follows:

[0095] Mean: If the feature space is Euclidean, the cluster center can be simply obtained by calculating the arithmetic mean of the coordinates of all sample feature vectors.

[0096] Mode: For discrete features, the cluster center may be the feature combination with the highest frequency, that is, the mode.

[0097] Centroid: Especially in clustering algorithms like k-means, the center of the cluster is also called the centroid, which is obtained by minimizing the sum of the squared distances of all points to the center through an iterative process.

[0098] The "preset spatial distribution" refers to a defined feature space, where each dimension corresponds to a specific feature of the image. The calculation of the "cluster center" is to capture the central tendency of sample features within each known category, thereby providing a compact and representative point for subsequent similarity calculation and deviation evaluation.

[0099] By summarizing the deviation values ​​of all categories, an overall deviation value that can reflect the overall difference between the image to be detected and all known categories is calculated. Select the appropriate summary processing method according to specific needs, including: Minimum value: select the minimum value among all deviation values. Average value: calculate the average value of all deviation values. Weighted sum: weighted sum of deviation values ​​of different categories.

[0100] The formula for calculating the deviation value between each spatial distribution feature and the regional feature using the distance measurement function is as follows:

[0101]

[0102] Where E[·] represents the mathematical expectation, that is, the average value of a random variable or function, σ(x,b) represents a deviation measurement function under a condition, x represents the image to be detected, b represents the identified region of interest (such as the lesion area), and the deviation measurement function measures the similarity or deviation between an initial region of the image to be detected and a known category, Y in Represents the category label of the known category (label within the "in-distribution" distribution), which is used to calculate the conditional expectation, E P [·] represents the expected value of the distribution feature P, P is the distribution feature of the cluster center of the known category, H represents the distance measurement function, Represents the local features of the initial region cropped from the image to be detected.

[0103] In this embodiment, the distribution features P of known categories are obtained in the following manner: the names of categories in the known category dataset are used to generate text inputs (e.g., "a photo of {label}"), and the text encoder T(·) processes these text inputs to generate class-specific features. The collection of features forms the distribution features of the known categories (i.e., ID space P) = {T(t i )|i=1,2,…,K}.

[0104] In step S3, in order to solve the problem that the prior art tends to misclassify unknown categories as the closest known categories, even if the prediction is inaccurate. This method may lead to false alarms and mislead doctors or other users. The present invention can quantify the difference between the image to be detected and all known categories by introducing a distance measurement function. If the overall deviation value exceeds the preset threshold, it indicates that the sample is likely not to belong to any known category, but to an OOD (Out-of-Distribution) sample or an abnormal situation. It can make more reasonable judgments when facing unseen data, rather than simply classifying it into the most similar known category, thereby reducing the false alarm rate.

[0105] S4. Compare the overall deviation value with a preset threshold value, determine whether the category to which the initial area belongs is an unknown category or a known category according to the comparison result, and mark the corresponding category label to respond to the request for anomaly detection.

[0106] In this embodiment, one or more thresholds are set for each known category based on historical data, experimental results, or the experience of domain experts. The preset threshold is used to determine whether the overall deviation value is within an acceptable range.

[0107] Compare the overall deviation value with the preset threshold to determine which known category best matches the image to be detected, or whether the image belongs to an unknown category, including: Based on the comparison results, use the best matching category as the label of the initial region. If no match is found, mark the initial region as an unknown category. Create a response object containing the category label and other relevant information (such as confidence, suggestions, etc.) in response to the anomaly detection request.

[0108] The whole process of the present invention is described in detail through an abnormal detection of autonomous driving in a specific insurance claim scenario:

[0109] When the autonomous vehicle monitoring system receives an anomaly detection request for an image of a new type of deer taken by the on-board camera, the global image encoder of the detection model analyzes the image to obtain global features such as roads, trees, and sky. The local image encoder analyzes the image to obtain regional features such as the unique markings and huge size of the new type of deer, and the global features and regional features are fused into fused features.

[0110] Then, known animal samples (such as deer, cattle, sheep, etc.) and their features are obtained from the database, and the similarity between the fusion features and the features of each sample is calculated. If the similarity with deer is high, the maximum value is taken from the similarities of each known category. Then, a distance measurement function is constructed to calculate the cluster center of the known category sample features, and the overall deviation value between the regional features and the cluster center in the preset space is calculated. Finally, the overall deviation value is compared with the threshold. If the deviation value is greater than the threshold, the initial area (new deer) is determined to be an unknown category and marked; if it is less than or equal to the threshold, it is a known category and marked, in response to the abnormal detection request. The vehicle is slowed down to reduce the occurrence of insurance claims.

[0111] The whole process of the present invention is described in detail by a specific medical example (lesion detection in lung CT scan):

[0112] The doctor uploaded a CT scan image of the patient's lungs to check for potential lung lesions (such as tumors, pneumonia, etc.).

[0113] After receiving the image, it is first preprocessed (resizing, normalizing pixel values). Then, using a pre-trained global image encoder (such as ResNet), high-level semantic information of the entire CT image is gradually extracted through multi-layer convolution and pooling operations to generate global features representing the entire image.

[0114] Next, we use an object detection model (such as Faster R-CNN) to automatically identify possible lesion areas and crop these initial areas from the original image. Then we load the local image encoder to perform forward propagation on each initial area to extract the local features of the area.

[0115] The global features and local features are combined to form fusion features. This step ensures that the final feature representation contains both overall information and local details, improving the accuracy of subsequent analysis.

[0116] Obtain sample features of all known disease types from the preset database (e.g. different types of lung diseases, including pneumonia, lung cancer, etc.). For each known category, traverse all its sample feature vectors and use the preset similarity measurement method (e.g. cosine similarity, Euclidean distance, etc.) to calculate the similarity between the fusion feature and each sample feature in each category.

[0117] For each known category, all similarity results calculated are sorted from largest to smallest, and the maximum value is selected as the maximum similarity value under that category. This step helps determine which known category the image to be detected is most likely to belong to.

[0118] A distance measurement function is constructed based on the maximum similarity value. The function receives the fused features of the image to be detected and the category label of the known category as input, and returns the corresponding negative maximum similarity value as the distance.

[0119] Use this distance measurement function to calculate the deviation between the regional features of the image to be detected and the distribution features of the cluster centers of known categories. The distribution features can be the average value or cluster center of all sample feature vectors under the category. Evaluate the overall difference between the initial region and all known categories.

[0120] The calculated overall deviation value is compared with the preset threshold. If the deviation value is greater than the threshold, the initial region is considered to belong to the unknown category, indicating that a new or rare lesion may exist; otherwise, it is classified as one of the closest known categories.

[0121] Based on the judgment result, the system will mark the category label accordingly and create a response object containing the category label and other relevant information (such as confidence, recommendation, etc.) in response to the anomaly detection request. For example, if the test result shows that the lesion is most likely lung cancer, the system will mark the area in the image and attach the corresponding diagnostic recommendation for the doctor's reference.

[0122] In one embodiment, judging whether the category to which the initial region belongs is an unknown category or a known category according to the comparison result, and marking a corresponding category label, includes:

[0123] If the comparison result is that the overall deviation value is greater than the preset threshold, it is determined that the initial area belongs to the unknown category;

[0124] If the comparison result is that the overall deviation value is less than or equal to the preset threshold, it is determined that the initial area belongs to a known category;

[0125] The initial region is marked with a category label according to the category to which it belongs.

[0126] The overall deviation value and the preset threshold are input into the preset decision function to determine whether the initial area of ​​the image to be tested is an unknown sample (OOD sample). The decision function formula is as follows:

[0127]

[0128] in, represents a decision function, represents the local features of the initial region, Y in Represents the category label of the known category (label within the "in-distribution" distribution), which is used to calculate the conditional expectation, E[σ(x,b)|Y in ] indicates that given the input data, the average similarity or deviation between the initial region and all known categories is calculated. γ indicates the threshold value used to distinguish known categories (ID) from unknown categories (OOD). ID indicates that the initial region belongs to a known category, and OOD indicates that the initial region does not belong to any known category and is an unknown category.

[0129] In step S4, the overall deviation value is used as a clear quantitative indicator to provide a clear basis for decision-making. When the deviation value exceeds the set range, the image to be detected can be marked as an unknown category and prompted to require further professional examination, which not only improves the safety and reliability of diagnosis, but also promotes the research and understanding of new or rare diseases.

[0130] In steps S1-S4, the present invention extracts global features and regional features of the initial region from the image to be detected at the same time and fuses the two to more accurately understand the image content. The fused features are compared with the sample features of all known categories in the database to find the most similar category, and a preliminary assessment can be made as to whether the image to be detected belongs to any known category.

[0131] A distance measurement function for quantifying deviations is constructed based on the maximum similarity value. The distance measurement function can determine the degree of difference between the new sample and the known category, that is, whether the new sample may be an unknown category. The overall deviation in the spatial distribution between the initial regional features of the suspected anomaly and each cluster center (representing the known category) is calculated. The overall deviation value reflects the degree to which the new sample deviates from the known category distribution and is a key indicator for determining whether the sample is OOD.

[0132] Finally, the overall deviation value is compared with the preset threshold to determine whether the detected area belongs to a known category or an unknown category. If the deviation value exceeds the threshold, the area is considered to be an unknown category, indicating that there may be a new, unseen type of lesion; otherwise, it is classified as a known category.

[0133] The present invention not only improves the recognition accuracy of known categories, but also enhances the sensitivity to unknown categories or abnormal situations. It makes classification decisions by integrating global and local features, quantifying similarities and deviation values, and setting reasonable thresholds. It ultimately achieves safer and more reliable medical image analysis, which is particularly suitable for the early detection and research of complex lesions in the health care field.

[0134] like Figure 3 FIG. 1 is a schematic diagram of a module of an image anomaly detection device provided by an embodiment of the present invention.

[0135] The image anomaly detection device 100 of the present invention can be installed in a device. According to the functions to be implemented, the image anomaly detection device 100 can include a fusion module 110, a selection module 120, a calculation module 130 and a marking module 140. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a device processor and can complete fixed functions, which are stored in the memory of the device.

[0136] In this embodiment, the functions of each module / unit are as follows:

[0137] A fusion module 110 is used to receive a request for abnormality detection on an image to be detected, extract global features of the image to be detected using a global image encoder of a detection model, extract regional features of an initial region of the image to be detected using a local image encoder of the detection model, and fuse the global features and the regional features to obtain fused features;

[0138] A selection module 120 is used to obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set;

[0139] A calculation module 130 is used to construct a distance measurement function according to each maximum similarity value, calculate the cluster centers of all sample features in each known category in a preset spatial distribution, and use the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center in the spatial distribution;

[0140] The marking module 140 is used to compare the overall deviation value with a preset threshold, determine whether the category to which the initial area belongs is an unknown category or a known category according to the comparison result, and mark the corresponding category label to respond to the request for anomaly detection.

[0141] In one embodiment, the fusion module 110 is specifically configured to:

[0142] Adjusting the size of the image to be detected and performing normalization processing on the image to be detected to obtain a preprocessed image;

[0143] The global image encoder is used to sequentially perform forward propagation, feature map extraction and feature aggregation processing on the preprocessed image to obtain the global feature.

[0144] In one embodiment, the fusion module 110 is specifically configured to:

[0145] Using the target detection model to identify the initial area of ​​the image to be detected;

[0146] The local image encoder is used to sequentially perform forward propagation, feature map extraction and feature aggregation processing on the initial region to obtain the regional features.

[0147] In one embodiment, the selection module 120 is specifically configured to:

[0148] Traverse the similarity result set of each known category and sort the similarity result of each known category in descending order;

[0149] Select the maximum similarity value among each known category according to the sorting order.

[0150] In one embodiment, the calculation module 130 is specifically configured to:

[0151] The maximum similarity value among each known category is stored in a preset dictionary;

[0152] The keys of the dictionary are set to be the category labels of each known category, and the values ​​of the dictionary are set to be the maximum similarity values ​​of each known category;

[0153] The set dictionary is updated to obtain the distance measurement function.

[0154] In one embodiment, the calculation module 130 is specifically configured to:

[0155] Calculating the deviation value between the distribution characteristics of each cluster center and the regional characteristics by using the distance measurement function to obtain the deviation value between the initial region and each known category;

[0156] Aggregation processing is performed on all deviation values ​​to obtain the overall deviation value, wherein the aggregation processing includes taking a minimum value, an average value, or a weighted sum.

[0157] In one embodiment, the marking module 140 is specifically configured to:

[0158] If the comparison result is that the overall deviation value is greater than the preset threshold, it is determined that the initial area belongs to the unknown category;

[0159] If the comparison result is that the overall deviation value is less than or equal to the preset threshold, it is determined that the initial area belongs to a known category;

[0160] The initial region is marked with a category label according to the category to which it belongs.

[0161] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an image anomaly detection method.

[0162] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of an image anomaly detection method are implemented.

[0163] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0164] Receiving a request for abnormality detection on an image to be detected, extracting global features of the image to be detected using a global image encoder of a detection model, extracting regional features of an initial region of the image to be detected using a local image encoder of the detection model, and fusing the global features and the regional features to obtain fused features;

[0165] Obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set;

[0166] Constructing a distance measurement function according to each maximum similarity value, calculating the cluster center of all sample features in each known category on a preset spatial distribution, and using the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center on the spatial distribution;

[0167] The overall deviation value is compared with a preset threshold value, and the category to which the initial area belongs is determined as an unknown category or a known category according to the comparison result, and a corresponding category label is marked to respond to the request for anomaly detection.

[0168] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0170] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0171] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit it. If software tools or components other than those of the company appear in the application embodiments, they are only used for illustrative purposes and do not represent actual use. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for detecting anomalies in an image, characterized in that: The method comprises: Receiving a request for abnormality detection on an image to be detected, extracting global features of the image to be detected using a global image encoder of a detection model, extracting regional features of an initial region of the image to be detected using a local image encoder of the detection model, and fusing the global features and the regional features to obtain fused features; Obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set; Constructing a distance measurement function according to each maximum similarity value, calculating the cluster center of all sample features in each known category on a preset spatial distribution, and using the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center on the spatial distribution; The overall deviation value is compared with a preset threshold value, and the category to which the initial area belongs is determined as an unknown category or a known category according to the comparison result, and a corresponding category label is marked to respond to the request for anomaly detection.

2. The image anomaly detection method according to claim 1, characterized in that: The global image encoder using a preset detection model extracts the global features of the image to be detected, including: Adjusting the size of the image to be detected and performing normalization processing on the image to be detected to obtain a preprocessed image; The global image encoder is used to sequentially perform forward propagation, feature map extraction and feature aggregation processing on the preprocessed image to obtain the global feature.

3. The image anomaly detection method according to claim 1, characterized in that: The extracting the regional features of the initial region of the image to be detected by using the local image encoder of the detection model includes: Using the target detection model to identify the initial area of ​​the image to be detected; The local image encoder is used to sequentially perform forward propagation, feature map extraction and feature aggregation processing on the initial region to obtain the regional features.

4. The image anomaly detection method according to claim 1, characterized in that: The step of selecting the maximum similarity value in the corresponding known category from each similarity result set includes: Traverse the similarity result set of each known category and sort the similarity result of each known category in descending order; Select the maximum similarity value among each known category according to the sorting order.

5. The image anomaly detection method according to claim 1, characterized in that: The step of constructing a distance measurement function according to each maximum similarity value includes: The maximum similarity value among each known category is stored in a preset dictionary; The keys of the dictionary are set to be the category labels of each known category, and the values ​​of the dictionary are set to be the maximum similarity values ​​of each known category; The set dictionary is updated to obtain the distance measurement function.

6. The image anomaly detection method according to claim 1, wherein: The using the distance measurement function to calculate the overall deviation value of the distribution characteristics of the regional characteristics and each cluster center in the spatial distribution includes: Calculating the deviation value between the distribution characteristics of each cluster center and the regional characteristics by using the distance measurement function to obtain the deviation value between the initial region and each known category; Aggregation processing is performed on all deviation values ​​to obtain the overall deviation value, wherein the aggregation processing includes taking a minimum value, an average value, or a weighted sum.

7. The image anomaly detection method according to claim 1, characterized in that: The step of determining the category to which the initial region belongs and marking the category label according to the comparison result includes: If the comparison result is that the overall deviation value is greater than the preset threshold, it is determined that the initial area belongs to the unknown category; If the comparison result is that the overall deviation value is less than or equal to the preset threshold, it is determined that the initial area belongs to a known category; The initial region is marked with a category label according to the category to which it belongs.

8. An image anomaly detection device, characterized in that: The device comprises: A fusion module is used to receive a request for abnormality detection on an image to be detected, extract global features of the image to be detected using a global image encoder of a detection model, extract regional features of an initial region of the image to be detected using a local image encoder of the detection model, and fuse the global features and the regional features to obtain fused features; A selection module is used to obtain samples of all known categories for training the detection model from a database, extract sample features of each sample, calculate the similarity between the fused feature and each sample feature, obtain a similarity result set between the fused feature and each known category, and select the maximum similarity value in the corresponding known category from each similarity result set; A calculation module is used to construct a distance measurement function according to each maximum similarity value, calculate the cluster center of all sample features in each known category on a preset spatial distribution, and use the distance measurement function to calculate the overall deviation value of the distribution features of the regional features and each cluster center on the spatial distribution; The marking module is used to compare the overall deviation value with a preset threshold value, determine whether the category to which the initial area belongs is an unknown category or a known category according to the comparison result, and mark the corresponding category label to respond to the request for anomaly detection.

9. A device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores an image anomaly detection program executable by the at least one processor, and the image anomaly detection program is executed by the at least one processor so that the at least one processor can perform the image anomaly detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: An image anomaly detection program is stored on the computer-readable storage medium, and the image anomaly detection program can be executed by one or more processors to implement the image anomaly detection method according to any one of claims 1 to 7.

Images