Abnormal sample detection method and device, equipment and storage medium
By performing feature extraction and similarity matrix processing on the current image frame and its above frame in medical video, uncertain values are calculated to judge abnormal samples, the problem of existing models identifying abnormal samples in medical video diagnosis is solved, and higher diagnostic reliability and accuracy are achieved.
Patent Information
- Application Number
- CN202411732693.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-07
AI Technical Summary
Existing object detection models are difficult to identify abnormal samples with high accuracy in medical images, especially in uneven data distribution and video diagnosis scenarios, and existing models cannot effectively handle abnormal situations in medical videos.
By acquiring the current image frame and several frames above, feature extraction is performed to generate candidate features, calculate the similarity matrix of the feature vector, process the matrix to obtain the feature to be predicted, and calculate its uncertain value to determine whether it is an abnormal sample.
This method can more accurately identify abnormal samples in medical videos, improve the reliability and accuracy of medical video diagnosis, and reduce the probability of misdiagnosis and missed diagnosis.
Smart Images
Figure CN119579997A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of medical video diagnosis technology, and specifically relates to an abnormal sample detection method, device, equipment and storage medium. Background Art
[0002] AI healthcare refers to the application of artificial intelligence technology in the medical field, including medical diagnosis, treatment, disease prediction, medical image analysis, drug development and many other aspects, in order to improve medical efficiency, improve medical quality and reduce medical costs. The technological development of AI healthcare has provided the possibility for automated and precise analysis of medical images.
[0003] In medical image data, the number of normal samples usually far exceeds the number of abnormal samples, and for some rare diseases, the number of image samples is very small, and the existing target detection models require enough sample training to learn effective features. Therefore, it is difficult to train a model with high accuracy to identify these abnormal samples, resulting in out-of-distribution (OOD) data that is difficult to handle in practical applications. In addition, in medical scenarios, the development of many diseases is a dynamic process. When doctors view a large number of medical videos, they may miss some key abnormalities due to visual fatigue or difficulty in paying attention to multiple details at the same time. Most of the existing target detection models can only detect a single image and cannot detect medical videos. Therefore, it is necessary to improve the existing target detection models so that they can detect videos and infer abnormal samples that appear unreasonably in the current image frame. Summary of the invention
[0004] The present application provides an abnormal sample detection method, apparatus, device and storage medium, which can fully mine the uncertain information on the current image frame, and then infer possible abnormal samples, thereby improving the reliability of medical video diagnosis.
[0005] In order to solve the above technical problems, in a first aspect, the present application provides an abnormal sample detection method, comprising the following steps:
[0006] Acquire a current image frame and a number of previous image frames located before the current image frame;
[0007] Performing feature extraction on the current image frame and a plurality of the previous image frames to generate candidate features corresponding to the current image frame and each of the previous image frames;
[0008] Generate a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames;
[0009] Generate a similarity matrix based on the similarity between the current feature vector and each of the above frame feature vectors, and process the similarity matrix to obtain the feature to be predicted corresponding to the current image frame;
[0010] An uncertainty value of the feature to be predicted is calculated, and based on the uncertainty value, it is determined whether the feature to be predicted is an abnormal sample.
[0011] As a further improvement of the present application, the acquiring of the current image frame and a plurality of previous image frames located before the current image frame includes:
[0012] Acquire a current image frame and a plurality of consecutive previous image frames located before the current image frame;
[0013] Alternatively, a current image frame and several consecutive previous image frames before the current image frame are acquired, the several consecutive previous image frames are screened, and similar previous image frames among the several consecutive previous image frames are filtered out.
[0014] As a further improvement of the present application, the step of extracting features from the current image frame and the plurality of previous image frames to generate candidate features corresponding to the current image frame and each of the previous image frames includes:
[0015] Inputting the current image frame and several of the above image frames into a target detection model, performing feature extraction on the current image frame and several of the above image frames through a feature extractor in the target detection model, and generating deep features corresponding to the current image frame and each of the above image frames;
[0016] Processing the deep features corresponding to the current image frame and each of the above image frames by the candidate feature generator in the target detection model to obtain the candidate features corresponding to the current image frame and each of the above image frames;
[0017] Among them, a number of candidate features are generated corresponding to the current image frame and each of the above image frames.
[0018] As a further improvement of the present application, the generating of the current frame feature vector and the previous frame feature vector corresponding to each of the previous image frames based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames includes:
[0019] The candidate features corresponding to the current image frame and the candidate features corresponding to each of the above image frames are encoded by a feature encoder to obtain a current frame feature vector corresponding to the current image frame. And the above frame feature vector corresponding to each of the above image frames j ranges from 1 to i-1;
[0020] Among them, i is the current image frame, and j is a number of previous image frames.
[0021] As a further improvement of the present application, the calculating the similarity between the current feature vector and each of the above frame feature vectors to generate a similarity matrix, and processing the similarity matrix to obtain the feature to be predicted corresponding to the current image frame, includes:
[0022] according to Calculating the similarity between the current image frame and each of the previous image frames respectively, to obtain a similarity matrix between the current image frame and the plurality of previous image frames;
[0023] The similarity matrix is input into the feature encoder to obtain the to-be-predicted features corresponding to the current image frame.
[0024] As a further improvement of the present application, the step of calculating the uncertainty value of the current image frame according to the feature to be predicted includes:
[0025] The features to be predicted are input into the classification prediction head, according to the formula Calculate the uncertainty value of the feature to be predicted; where f k It is the probability value of the kth class output by the classification prediction head.
[0026] As a further improvement of the present application, judging whether the feature to be predicted is an abnormal sample based on the uncertainty value includes:
[0027] The uncertainty value of the feature to be predicted is compared with a preset threshold value. When the uncertainty value is greater than the preset threshold value, the feature to be predicted is judged to be an abnormal sample; otherwise, the feature to be predicted is judged to be a normal sample.
[0028] In a second aspect, the present application provides an abnormal sample detection device, comprising:
[0029] An acquisition module, which acquires a current image frame and a plurality of previous image frames located before the current image frame;
[0030] A first generating module performs feature extraction on the current image frame and a plurality of the above image frames to generate candidate features corresponding to the current image frame and each of the above image frames;
[0031] A second generating module, based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames, generates a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames;
[0032] A calculation module, which generates a similarity matrix based on the similarity between the current feature vector and each of the above frame feature vectors, and processes the similarity matrix to obtain the feature to be predicted corresponding to the current image frame;
[0033] The judgment module calculates the uncertainty value of the feature to be predicted, and judges whether the feature to be predicted is an abnormal sample based on the uncertainty value.
[0034] In a third aspect, the present application provides a computer device, which includes a processor and a memory coupled to the processor, wherein a computing program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned abnormal sample detection methods.
[0035] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any of the above-mentioned abnormal sample detection methods.
[0036] Compared with the prior art, the abnormal sample detection method provided by the present application generates corresponding candidate features by extracting features from the current image frame and several previous image frames located before the current image frame, generates a current frame feature vector and a previous frame feature vector corresponding to each previous image frame based on the candidate features of each frame, generates a similarity matrix according to the similarity calculation between the current feature vector and each previous frame feature vector, processes the similarity matrix to obtain the feature to be predicted, calculates the uncertainty value of the feature to be predicted to determine whether the feature to be predicted is an abnormal sample, and infers whether the feature to be predicted in the current image frame is an abnormal sample by combining the current image frame and the previous image frame located before the current image frame for judgment, which can fully mine the uncertain information on the current image frame and further infer the possible abnormal samples; it can also be applied to the field of medical video diagnosis, can accurately identify abnormal features different from normal situations, help doctors more accurately determine the presence or absence of diseases and the type of diseases, and provide a reliable basis for subsequent precise treatment; it can analyze medical videos frame by frame, detect abnormal samples with consistent and objective standards, effectively reduce the probability of misdiagnosis and missed diagnosis, and improve the overall quality of medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 A flowchart of an abnormal sample detection method provided in an embodiment of the present application;
[0039] Figure 2 A schematic diagram of the structure of the abnormal sample detection method provided in the embodiment of the present application;
[0040] Figure 3 A schematic diagram of the structure of a similarity matrix in the abnormal sample detection method provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of an abnormal sample detection device provided in an embodiment of the present application;
[0042] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;
[0043] Figure 6 A schematic diagram of the structure of the storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solution and advantages of the embodiment of the present application more clear, the embodiment of the present application is further described in detail below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the embodiment of the present application and are not used to limit the embodiment of the present application.
[0045] In the description of the embodiments of the present application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically limited. In the embodiments of the present application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.
[0046] In order to make the description of the disclosed content more detailed and complete, the following is an illustrative description of the implementation methods and specific embodiments of the embodiments of the present application; however, this is not the only form of implementing or using the specific embodiments of the embodiments of the present application. The implementation methods cover the features of multiple specific embodiments and the method steps and sequences used to construct and operate these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and step sequences.
[0047] In the embodiments of the present application, "exemplary", "in some embodiments", "in another embodiment", etc. are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.
[0048] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0049] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0050] The technological development of AI medicine has made it possible to automatically and accurately analyze medical images. However, in medical image data, the number of normal samples is usually much larger than the number of abnormal samples, and for some rare diseases, the number of image samples is very small. Model training requires enough samples to learn effective features. Therefore, it is difficult to train a highly accurate model to identify these abnormalities, resulting in the widespread use of abnormal sample distribution detection.
[0051] In the field of AI medicine, abnormal sample distribution detection can effectively identify samples that do not meet standard specifications. Especially in telemedicine scenarios, it can help doctors diagnose and treat diseases more accurately and improve medical standards.
[0052] Since existing target detection models have not been exposed to samples of unknown types during training, they can make accurate judgments in trained known environments (in-distribution, ID). However, in practical applications, they often encounter out-of-distribution (OOD) data that is difficult to handle. In medical scenarios, the development of many diseases is a dynamic process. When doctors view a large number of medical videos, they may miss some key abnormal situations due to visual fatigue or difficulty in paying attention to multiple details at the same time. Most of the existing target detection models can only detect a single image and cannot detect medical videos. Therefore, it is necessary to improve the existing target detection models so that they can detect videos and infer abnormal samples that appear unreasonably in the current image frame.
[0053] In view of this, please refer to Figure 1-Figure 6 The embodiments of the present application provide an abnormal sample detection method, device, apparatus and storage medium, which can fully mine the uncertain information on the current image frame, and then infer possible abnormal samples, thereby improving the reliability of medical video diagnosis.
[0054] Please refer to Figure 1 , is a flow chart of an abnormal sample detection method provided in an embodiment of the present application, the detection method comprising the following steps:
[0055] Step S1: acquiring a current image frame and a plurality of previous image frames located before the current image frame;
[0056] As an optional implementation manner, the above-mentioned acquiring the current image frame and a plurality of previous image frames located before the current image frame includes:
[0057] Acquire a current image frame and a plurality of consecutive previous image frames located before the current image frame;
[0058] Alternatively, a current image frame and several consecutive previous image frames before the current image frame are obtained, and the several consecutive previous image frames before the current image frame are screened to filter out similar previous image frames among the several consecutive previous image frames.
[0059] In an embodiment of the present application, taking medical video as an example, it is necessary to obtain the image frame of the current medical video and several previous frames located before the current image frame. Taking the current image frame as the i-th frame as an example, it is necessary to obtain the current image frame of the i-th frame and the previous image frames of the 1st to i-1th frames located before the current image frame.
[0060] Preferably, the current image frame and several consecutive previous image frames before the current image frame may be acquired.
[0061] Alternatively, the current image frame and several consecutive previous image frames before the current image frame may be obtained, and the several consecutive previous image frames may be screened to remove the previous image frames with higher similarity, including key frames in the previous image frames.
[0062] The above several consecutive above image frames, or only retaining the key frames in the above image frames are both feasible, and can be selected according to the specific number of the above image frames, or the actual detection accuracy requirements. This application does not impose any restrictions on this.
[0063] Step S2: extracting features from the current image frame and a plurality of the previous image frames to generate candidate features corresponding to the current image frame and each of the previous image frames;
[0064] As an optional implementation manner, feature extraction is performed on the current image frame and several of the above image frames to generate candidate features corresponding to the current image frame and each of the above image frames, including:
[0065] Inputting the current image frame and several of the above image frames into a target detection model, performing feature extraction on the current image frame and several of the above image frames through a feature extractor in the target detection model, and generating deep features corresponding to the current image frame and each of the above image frames;
[0066] Processing the deep features corresponding to the current image frame and each of the above image frames by the candidate feature generator in the target detection model to obtain the candidate features corresponding to the current image frame and each of the above image frames;
[0067] Among them, a number of candidate features are generated corresponding to the current image frame and each of the above image frames.
[0068] In an embodiment of the present application, the current image frame and several previous image frames need to be input into a target detection model, and the current image frame and several previous image frames are extracted by a feature extractor in the target detection model to generate deep features corresponding to the current image frame and each previous image frame.
[0069] In the target detection model, the feature extractor backbone is a key component that can extract deep features from the input image frame. These deep features can effectively describe the target object in the current image frame. For example, in medical image detection, such as detecting fractures in X-rays or tumors in CT images, the feature extractor backbone can extract the shape and texture of the target area, providing basic information for subsequent target positioning and classification.
[0070] Of course, in actual applications, extracting the current image frame and several previous image frames through the feature extractor in the target detection model may result in a large amount of computation. Therefore, several previous image frames can be input in advance during training, and the feature extractor can be used to extract deep features of the input several previous image frames in advance. When the actual detection is performed, the feature extractor can be used to generate the deep features corresponding to the current image frame.
[0071] For further information, please refer to Figure 2 , which is a structural diagram of the abnormal sample detection method provided in an embodiment of the present application. After generating the deep features of the current image frame and several previous image frames, the deep features corresponding to the current image frame and each previous image frame are processed by the candidate feature generator in the target detection model to obtain the candidate features corresponding to the current image frame and each previous image frame.
[0072] It can be understood that the current image frame and each of the above image frames have their own corresponding candidate features generated, and the current image frame and each of the above image frames have several candidate features generated, that is, each image frame should have several candidate features.
[0073] The candidate feature generator, namely the Propsal generator (Region Proposal Network, RPN), can quickly screen out a series of candidate regions where targets may exist in the image, greatly reducing the search range in the subsequent detection stage and improving the efficiency of target detection. For example, in medical images such as X-rays and CT scans that may contain various organs and various pathological tissues of the human body, the Proposal generator can help locate areas that may be lesions, providing a basis for further accurate detection.
[0074] This application combines the Propsal generator with the feature extractor backbone, so that the feature extractor backbone inputs the extracted deep features into the Propsal generator, and performs multi-level fusion of the input deep features inside the Propsal generator, so that the Propsal generator uses richer feature information to generate more accurate candidate regions, thereby obtaining the candidate features corresponding to the current image frame and each previous image frame.
[0075] Step S3: generating a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames;
[0076] As an optional implementation manner, the above-mentioned generating a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames includes:
[0077] The candidate features corresponding to the current image frame and the candidate features corresponding to each of the above image frames are encoded by a feature encoder to obtain a current frame feature vector corresponding to the current image frame. And the above frame feature vector corresponding to each of the above image frames j ranges from 1 to i-1;
[0078] Among them, i is the current image frame, and j is a number of previous image frames.
[0079] In the embodiment of the present application, after the feature extractor backbone and the Propsal generator generate candidate features corresponding to the current image frame and each previous image frame, the candidate features corresponding to each image frame are input into the feature encoder, and the feature encoder encodes the candidate features corresponding to the current image frame and each previous image frame to obtain the current frame feature vector corresponding to the current image frame. And the above frame feature vector corresponding to each of the above image frames
[0080] Among them, the feature encoder in the target detection model is mainly responsible for converting candidate features into representative and discriminative feature representations. For example, in medical image target detection, the feature encoder can extract information such as the contour features of organs and texture differences in lesion areas from X-ray, CT or MRI images to help the model accurately find the target and determine its category.
[0081] Common feature encoders can be implemented based on CNN (Convolutional Neural Networks). The candidate features of the current image frame and each of the previous frames are input into the feature encoder to generate the feature vector corresponding to each frame without interfering with each other, that is, the current frame feature vector corresponding to the required current image frame is generated. And the above frame feature vector corresponding to each of the above image frames Wherein, i is the current image frame, j is a number of previous image frames, and the value of j ranges from 1 to i-1.
[0082] Please refer to Figure 3 , which is a structural diagram of the similarity matrix in the abnormal sample detection method provided in the embodiment of the present application. It can be observed that the left side of the figure is the current frame feature vector corresponding to the current image frame. The right side of the figure shows the feature vector of the previous frame corresponding to the previous image frame. to
[0083] Step S4: calculating the similarity between the current feature vector and each of the feature vectors of the previous frames to generate a similarity matrix, and processing the similarity matrix to obtain the feature to be predicted corresponding to the current image frame;
[0084] As an optional implementation, the above-mentioned calculation of the similarity between the current feature vector and each of the above frame feature vectors to generate a similarity matrix, and processing the similarity matrix to obtain the feature to be predicted corresponding to the current image frame, includes:
[0085] according to Calculating the similarity between the current image frame and each of the previous image frames respectively, to obtain a similarity matrix between the current image frame and the plurality of previous image frames;
[0086] The similarity matrix is input into the feature encoder to obtain the to-be-predicted features corresponding to the current image frame.
[0087] In the embodiment of the present application, the feature encoder generates a current frame feature vector corresponding to the current image frame. And the context frame feature vector corresponding to each context image frame Then, through the formula The similarity between the current image frame and each previous image frame is calculated respectively, and finally a similarity matrix is obtained.
[0088] Taking medical image target detection as an example, when analyzing an X-ray or CT image, the similarity matrix can reveal the similarities between different tissue regions, such as the similarities and differences in feature expressions between normal tissue and diseased tissue. Therefore, this similarity matrix is of great significance for the feature distribution of the current image frame and the subsequent extraction of features to be predicted.
[0089] Furthermore, the similarity matrix is input into the feature encoder again to obtain the features to be predicted of the current image frame, that is, the target prediction features of the current image frame.
[0090] Taking the CNN convolutional neural network as an example, the convolutional neural network processes the similarity matrix through convolutional layer operations, pooling layer cooperation and multi-layer architecture collaboration. The convolution layer operation slides on the similarity matrix and extracts local features through convolution operations. The pooling layer cooperation selects the maximum value in the similarity matrix as the output in a specific area. This can reduce the data dimension and highlight the most significant similarity features in the area. The multi-layer architecture collaboration extracts more advanced semantic features based on the convolutional layer operation and pooling layer cooperation.
[0091] Taking medical image target detection as an example, the features to be predicted may include precise location information of diseased tissue determined by location-related codes in the features, or preliminary judgment information of the lesion type determined based on codes in the features that reflect the similarity between the lesion features and known lesion types.
[0092] In practical applications, these features to be predicted will be sent to subsequent classification prediction components to accurately determine the category of the target in the image, such as whether it is a tumor or a cyst, and the location, such as determining the specific coordinate range of the tumor in the image, so as to achieve efficient and accurate target detection; this application does not go into too much detail on the detailed steps of how the convolutional neural network obtains the features to be predicted of the current image frame.
[0093] Step S5: Calculate the uncertainty value of the feature to be predicted, and determine whether the feature to be predicted is an abnormal sample based on the uncertainty value.
[0094] As an optional implementation, the feature to be predicted is input into the classification prediction head, according to Calculate the uncertainty value of the feature to be predicted; where f k The probability value of the kth class output by the classification prediction head;
[0095] The uncertainty value of the feature to be predicted is compared with a preset threshold value. When the uncertainty value is greater than the preset threshold value, the feature to be predicted is judged to be an abnormal sample; otherwise, the feature to be predicted is judged to be a normal sample.
[0096] In an embodiment of the present application, after the feature to be predicted is obtained, the feature to be predicted can be input into a subsequent classification prediction head, and the uncertainty of the feature to be predicted is output through the classification prediction head. Taking k classification as an example, it can be determined whether the feature to be predicted is k classification.
[0097] For example, in medical image detection, k categories may represent multiple disease types or tissue types. Taking the detection of lung CT images as an example, k categories can be normal tissue, pneumonia lesions, tuberculosis lesions, lung cancer lesions and other categories. The model extracts and processes the current image frame, and the features to be predicted obtained will be used to determine which of the k categories the target area belongs to, or whether it belongs to the k category. Usually, the features to be predicted will be input into the classification prediction head, which will map the features to be predicted to the probability distribution of k categories based on some classification algorithms, thereby realizing the k classification task.
[0098] Specifically, this application uses the formula Calculate the uncertainty value of the feature to be predicted, where f k It is the probability value of the kth class output by the classification prediction head.
[0099] In an optional embodiment, the classification prediction head can convert the unnormalized score output by the classification prediction head into a probability distribution based on a Softmax function, a Sigmoid function, or a Bayesian network, that is, the probability value of the kth class output by the classification prediction head.
[0100] It can be understood that in the target detection model, the unnormalized score output by the classification prediction head is the intermediate result obtained before the probability normalization. The fully connected layer in the classification prediction head will linearly change the input features to be predicted, and the output value generated is the unnormalized score. The unnormalized score output by the classification prediction head is then converted into a probability distribution through the above-mentioned Softmax function, Sigmoid function or Bayesian network, thereby obtaining the probability value of the kth class output by the classification prediction head.
[0101] In the embodiment of the present application, it is necessary to judge whether the feature to be predicted belongs to k categories. Of course, it is also possible to judge which specific category of the k categories the feature to be predicted belongs to. The present application does not impose too many restrictions on this.
[0102] Specifically, the uncertainty value calculated by the formula is compared with a preset threshold value. When the uncertainty value is greater than the preset threshold value, the feature to be predicted is judged to be an abnormal sample; otherwise, the feature to be predicted is judged to be a normal sample.
[0103] Taking the preset threshold of 0.5 as an example, when the calculated uncertainty value is greater than 0.5, the feature to be predicted is judged to be OOD out of distribution and is an abnormal sample. It can be understood that for K classification, samples that do not belong to K classification are considered abnormal samples out of distribution.
[0104] Of course, when training the target detection model in the early stage, normal video samples can be used for training. The key frames in the video samples, such as image frames with target features, can be input as single frames through manual identification, and the context frames before the key frames can be used as context image frames for training. In actual application, the image frames to be detected are input into the target detection model, and implemented according to the methods and steps provided by the above-mentioned abnormal sample detection method. The above training and detection methods are all feasible, and this application does not impose further restrictions on this.
[0105] The abnormal sample detection method provided in the present application extracts features from the current image frame and several previous image frames before the current image frame to generate corresponding candidate features, generates a current frame feature vector and a previous frame feature vector corresponding to each previous image frame based on the candidate features of each frame, generates a similarity matrix based on the similarity calculation between the current feature vector and each previous frame feature vector, processes the similarity matrix to obtain the feature to be predicted, calculates the uncertainty value of the feature to be predicted to determine whether the feature to be predicted is an abnormal sample, and infers whether the feature to be predicted in the current image frame is an abnormal sample by combining the current image frame and the previous image frames before the current image frame for judgment, which can fully mine the uncertain information on the current image frame and further infer the possible abnormal samples; it can also be applied to the field of medical video diagnosis, can accurately identify abnormal features different from normal conditions, help doctors more accurately determine the presence and type of diseases, and provide a reliable basis for subsequent precise treatment; it can analyze medical videos frame by frame, detect abnormal samples with consistent and objective standards, effectively reduce the probability of misdiagnosis and missed diagnosis, and improve the overall quality of medical diagnosis.
[0106] Based on the above abnormal sample detection method, this application provides an abnormal sample detection device, please refer to Figure 4 , is a schematic diagram of the structure of an abnormal sample detection device provided in an embodiment of the present application, the detection device includes an acquisition module 41, a first generation module 42, a second generation module 43, a calculation module 44 and a judgment module 45.
[0107] The acquisition module 41 is used to acquire a current image frame and a plurality of previous image frames located before the current image frame;
[0108] A first generating module 42, configured to extract features from the current image frame and a plurality of the previous image frames, and generate candidate features corresponding to the current image frame and each of the previous image frames;
[0109] A second generating module 43, configured to generate a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames;
[0110] A calculation module 44, configured to generate a similarity matrix based on the similarity between the current feature vector and each of the above frame feature vectors, and process the similarity matrix to obtain the feature to be predicted corresponding to the current image frame;
[0111] The judgment module 45 is used to calculate the uncertainty value of the feature to be predicted, and judge whether the feature to be predicted is an abnormal sample based on the uncertainty value.
[0112] As an optional implementation, the acquisition module 41 is used to acquire a current image frame and a plurality of consecutive previous image frames located before the current image frame;
[0113] Alternatively, a current image frame and several consecutive previous image frames before the current image frame are acquired, the several consecutive previous image frames are screened, and similar previous image frames among the several consecutive previous image frames are filtered out.
[0114] As an optional implementation, the first generation module 42 is used to input the current image frame and several of the previous image frames into the target detection model, perform feature extraction on the current image frame and several of the previous image frames through the feature extractor in the target detection model, and generate deep features corresponding to the current image frame and each of the previous image frames;
[0115] Processing the deep features corresponding to the current image frame and each of the above image frames by the candidate feature generator in the target detection model to obtain the candidate features corresponding to the current image frame and each of the above image frames;
[0116] Among them, a number of candidate features are generated corresponding to the current image frame and each of the above image frames.
[0117] As an optional implementation, the second generating module 43 is used to encode the candidate features corresponding to the current image frame and the candidate features corresponding to each of the above image frames through a feature encoder to obtain a current frame feature vector corresponding to the current image frame. And the above frame feature vector corresponding to each of the above image frames j ranges from 1 to i-1;
[0118] Among them, i is the current image frame, and j is a number of previous image frames.
[0119] As an optional implementation, the calculation module 44 is used according to Calculating the similarity between the current image frame and each of the previous image frames respectively, to obtain a similarity matrix between the current image frame and the plurality of previous image frames;
[0120] The similarity matrix is input into the feature encoder to obtain the to-be-predicted features corresponding to the current image frame.
[0121] As an optional implementation, the judgment module 45 is used to input the feature to be predicted into the classification prediction head, according to Calculate the uncertainty value of the feature to be predicted; where f k The probability value of the kth class output by the classification prediction head;
[0122] The uncertainty value of the feature to be predicted is compared with a preset threshold value. When the uncertainty value is greater than the preset threshold value, the feature to be predicted is judged to be an abnormal sample; otherwise, the feature to be predicted is judged to be a normal sample.
[0123] For other details about the technical solutions for implementing each module in the abnormal sample detection device provided in the above embodiment, please refer to the description of the abnormal sample detection method in the above embodiment, which will not be repeated here.
[0124] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0125] Please refer to Figure 5 , is a structural diagram of a computer device provided in an embodiment of the present application, wherein the computer device 50 includes a processor 51 and a memory 52 coupled to the processor 51.
[0126] The memory 52 stores a computer program. When the computer program is executed by the processor 51, the processor 51 executes the steps of the artificial intelligence-based actuarial temporary classification method in the above-mentioned embodiment.
[0127] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0128] Please refer to Figure 6 , a schematic diagram of the structure of the storage medium provided in the embodiment of the present application, the computer-readable storage medium of the embodiment of the present application stores a computer program 60, and the computer program 60 is executed by the processor 51 to implement the artificial intelligence-based actuarial temporary classification method in the above embodiment, wherein the computer program 60 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a computer device such as a computer, a server, a mobile phone, and a tablet. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
[0129] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0130] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.
[0131] The above embodiments are merely exemplary embodiments used to illustrate the principles of the embodiments of the present application, but the embodiments of the present application are not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the embodiments of the present application, and these modifications and improvements are also considered as the protection scope of the embodiments of the present application.
Claims
1. A method for detecting abnormal samples, characterized in that: The following steps are involved: Acquire a current image frame and a number of previous image frames located before the current image frame; Performing feature extraction on the current image frame and a plurality of the previous image frames to generate candidate features corresponding to the current image frame and each of the previous image frames; Generate a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames; Generate a similarity matrix based on the similarity between the current feature vector and each of the above frame feature vectors, and process the similarity matrix to obtain the feature to be predicted corresponding to the current image frame; An uncertainty value of the feature to be predicted is calculated, and based on the uncertainty value, it is determined whether the feature to be predicted is an abnormal sample.
2. The abnormal sample detection method according to claim 1, characterized in that: The acquiring of the current image frame and a plurality of previous image frames located before the current image frame comprises: Acquire a current image frame and a plurality of consecutive previous image frames located before the current image frame; Alternatively, a current image frame and several consecutive previous image frames before the current image frame are acquired, the several consecutive previous image frames are screened, and similar previous image frames among the several consecutive previous image frames are filtered out.
3. The abnormal sample detection method according to claim 1, characterized in that: The step of extracting features from the current image frame and the plurality of previous image frames to generate candidate features corresponding to the current image frame and each of the previous image frames includes: Inputting the current image frame and several of the above image frames into a target detection model, performing feature extraction on the current image frame and several of the above image frames through a feature extractor in the target detection model, and generating deep features corresponding to the current image frame and each of the above image frames; Processing the deep features corresponding to the current image frame and each of the above image frames by the candidate feature generator in the target detection model to obtain the candidate features corresponding to the current image frame and each of the above image frames; Among them, a number of candidate features are generated corresponding to the current image frame and each of the above image frames.
4. The abnormal sample detection method according to claim 1, characterized in that: The generating, based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames, a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames comprises: The candidate features corresponding to the current image frame and the candidate features corresponding to each of the above image frames are encoded by a feature encoder to obtain a current frame feature vector corresponding to the current image frame. And the above frame feature vector corresponding to each of the above image frames j ranges from 1 to i-1; Among them, i is the current image frame, and j is a number of previous image frames.
5. The abnormal sample detection method according to claim 4, characterized in that: The calculating the similarity between the current feature vector and each of the above frame feature vectors to generate a similarity matrix, and processing the similarity matrix to obtain the feature to be predicted corresponding to the current image frame, comprises: according to Calculating the similarity between the current image frame and each of the previous image frames respectively, to obtain a similarity matrix between the current image frame and the plurality of previous image frames; The similarity matrix is input into the feature encoder to obtain the to-be-predicted features corresponding to the current image frame.
6. The abnormal sample detection method according to claim 1, characterized in that: The calculating the uncertainty value of the current image frame according to the feature to be predicted includes: The features to be predicted are input into the classification prediction head, according to the formula Calculate the uncertainty value of the feature to be predicted; where f k It is the probability value of the kth class output by the classification prediction head.
7. The abnormal sample detection method according to claim 6, characterized in that: The determining whether the feature to be predicted is an abnormal sample based on the uncertainty value includes: The uncertainty value of the feature to be predicted is compared with a preset threshold value. When the uncertainty value is greater than the preset threshold value, the feature to be predicted is judged to be an abnormal sample; otherwise, the feature to be predicted is judged to be a normal sample.
8. An abnormal sample detection device, characterized in that: include: An acquisition module, which acquires a current image frame and a plurality of previous image frames located before the current image frame; A first generating module performs feature extraction on the current image frame and a plurality of the above image frames to generate candidate features corresponding to the current image frame and each of the above image frames; A second generating module, based on the candidate features corresponding to the current image frame and the candidate features corresponding to each of the previous image frames, generates a current frame feature vector and a previous frame feature vector corresponding to each of the previous image frames; A calculation module, which generates a similarity matrix based on the similarity between the current feature vector and each of the above frame feature vectors, and processes the similarity matrix to obtain the feature to be predicted corresponding to the current image frame; The judgment module calculates the uncertainty value of the feature to be predicted, and judges whether the feature to be predicted is an abnormal sample based on the uncertainty value.
9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein a computing program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the abnormal sample detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the abnormal sample detection method according to any one of claims 1 to 7.