Abnormality detection method and device for medical data, equipment and medium

Abstract

The invention relates to the technical field of anomaly detection, in particular to a medical data anomaly detection method and device, equipment and a medium, and the method comprises the steps that to-be-detected medical data is acquired, and the to-be-detected medical data comprises target text data and target image data; performing feature extraction on the target text data by using a pre-trained first feature extractor to obtain target text vector features; performing feature extraction on the target image data by using a pre-trained second feature extractor to obtain target image features; calculating the similarity between the target text vector feature and the target image feature; if the similarity is smaller than or equal to a preset threshold value, the to-be-detected medical data are abnormal, and if the similarity is larger than the preset threshold value, the to-be-detected medical data are normal. According to the technical scheme, anomaly detection can be accurately carried out on the multi-modal medical data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of anomaly detection, and in particular to a method, device, equipment and medium for anomaly detection of medical data. Background Art

[0002] Anomaly detection is the problem of finding patterns in data that do not conform to expected behavior. These patterns that do not conform to expected behavior are often referred to as outliers, outliers, inconsistent observations, anomalies, anomalies, sudden events, special cases, or contaminants in different application areas. Among them, outliers and outliers are the two most commonly used terms in the context of anomaly detection. Outliers usually refer to normal data that are far away from normal data. Although their characteristics are similar to those of outliers, they are still normal data, while outliers usually refer to abnormal data. Anomaly detection plays an important role in a wide range of applications such as credit card fraud detection, insurance or healthcare fraud detection, lesion detection in medicine, network security intrusion detection, safety-critical system fault detection, and military surveillance. The importance of anomaly detection lies in the fact that anomalies in data are often converted into important (even critical) actionable information in various application areas.

[0003] Multimodal data refers to a collection of data from multiple different sources or types (such as images, text, audio, video, sensor data, etc.), which are usually different forms of descriptions of the same thing. For example, a photo of birds, an audio of bird calls, and a text describing birds constitute multimodal data, or a Chinese medical text and medical images describing the same disease. In the real world, information often exists in multiple forms. By combining these different types of data, we can often obtain richer and more comprehensive information.

[0004] At present, for medical data, the commonly used anomaly detection scheme is to use a large amount of normal medical data and / or abnormal medical data of a single modality to train an anomaly recognition model, and use the anomaly recognition model to identify whether the data of a single modality is abnormal. For example, use a plurality of normal medical texts and / or abnormal medical texts to train an abnormal medical text recognition model, and use the abnormal medical text recognition model to identify whether a piece of medical text data is abnormal. The inventors have found that there is currently no anomaly detection scheme for multimodal medical data, so how to accurately detect anomalies in multimodal medical data is a problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the problems in the related art, the embodiments of the present disclosure provide a method, apparatus, device and medium for detecting anomalies in medical data.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for detecting anomalies in medical data, the method comprising:

[0007] Acquiring medical data to be detected, wherein the medical data to be detected includes target text data and target image data;

[0008] Using a pre-trained first feature extractor to extract features from the target text data to obtain target text vector features;

[0009] Using a pre-trained second feature extractor to extract features from the target image data to obtain target image features;

[0010] Calculating the similarity between the target text vector feature and the target image feature;

[0011] If the similarity is less than or equal to a predetermined threshold, the medical data to be detected is abnormal; if the similarity is greater than the predetermined threshold, the medical data to be detected is normal.

[0012] In a possible implementation, the method further includes:

[0013] Obtaining matching normal medical multimodal datasets and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer;

[0014] A feature extraction model is obtained by training using a normal medical multimodal data set and an abnormal data set matching an abnormality. The feature extraction model includes a first feature extractor and a second feature extractor. The first feature extractor is used to extract the feature from Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

[0015] In a possible implementation, the use of a normal matching medical multimodal dataset and an abnormal matching abnormal dataset to train a feature extraction model includes:

[0016] From matching normal medical multimodal datasets Sample m samples in

[0017] Using the first feature extractor T φ right Extract features to get Using the second feature extractor C θ right Extract features to get

[0018] The following formula is used to update the model parameter φ in the first feature extractor in the s+1th cycle: s+1 :

[0019]

[0020] The following formula is used to update the model parameter θ in the second feature extractor in the s+1th cycle: s+1 :

[0021]

[0022] The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor;

[0023] Among them, φ s Indicates the first feature extractor T in the sth cycle φ The parameters in θ s Indicates the second feature extractor C in the sth cycle θ The parameters in , η is the predetermined step size, represents the gradient with respect to φ, represents the gradient with respect to θ, S aa for and The similarity between ab for and The similarity between .

[0024] In a possible implementation, the first feature extractor includes a language model that converts word segments in text data into word vectors, and the second feature extractor includes a convolutional neural network.

[0025] In a possible implementation, the similarity is calculated using a Gaussian kernel function.

[0026] In a possible implementation manner, the predetermined threshold is S ii The minimum value of S ii is the i-th pair of normally matched multimodal data and The similarity between the text vector features and the image features extracted by the trained first feature extractor and the second feature extractor is used.

[0027] In a second aspect, the present disclosure provides a method for training a feature extraction model, including:

[0028] Obtaining matching normal medical multimodal datasets and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer;

[0029] A feature extraction model is obtained by training using a normal medical multimodal data set and an abnormal data set matching an abnormality. The feature extraction model includes a first feature extractor and a second feature extractor. The first feature extractor is used to extract the feature from Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

[0030] In a third aspect, an embodiment of the present disclosure provides a device for detecting abnormalities in medical data, the device comprising:

[0031] A data acquisition module is configured to acquire medical data to be detected, wherein the medical data to be detected includes target text data and target image data;

[0032] The feature extraction module is configured to use a pre-trained first feature extractor to perform feature extraction on the target text data to obtain target text vector features; and use a pre-trained second feature extractor to perform feature extraction on the image data to obtain target image features;

[0033] A calculation module, configured to calculate the similarity between the target text vector feature and the target image feature;

[0034] The detection module is configured to determine that if the similarity is less than or equal to a predetermined threshold, the medical data to be detected is abnormal, and if the similarity is greater than the predetermined threshold, the medical data to be detected is normal.

[0035] In a fourth aspect, an embodiment of the present disclosure provides a training device for a feature extraction model, comprising:

[0036] A dataset acquisition module is configured to acquire a matching normal medical multimodal dataset and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer;

[0037] The training module is configured to use a normal medical multimodal data set and an abnormal data set that matches an abnormality to train a feature extraction model, wherein the feature extraction model includes a first feature extractor and a second feature extractor, wherein the first feature extractor is used to extract the feature from the normal medical multimodal data set and the abnormal data set that matches the abnormality. Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

[0038] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the processor to implement a method as described in any one of the first aspect or the second aspect.

[0039] In a sixth aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure, on which computer instructions are stored. When the computer instructions are executed by a processor, a method as described in any one of the first aspect or the second aspect is implemented.

[0040] According to the technical solution provided by the embodiment of the present disclosure, after obtaining the medical data to be detected, a pre-trained first feature extractor can be used to perform feature extraction on the target text data to obtain target text vector features, and a pre-trained second feature extractor can be used to perform feature extraction on the image data to obtain target image features; the similarity between the target text vector features and the target image features is calculated, and when the similarity is less than or equal to a predetermined threshold, the medical data to be detected is determined to be abnormal, and when the similarity is greater than the predetermined threshold, the medical data to be detected is determined to be normal. In this way, the collaborative matching between text data and image data of different modalities is taken into account, and normal medical data is avoided from being misjudged as abnormal medical data due to the text data and image data being misjudged as mismatched, so that anomaly detection is accurately performed on multi-modal medical data.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0043] Figure 1 A flowchart of a method for detecting anomalies in medical data provided by an embodiment of the present disclosure is shown.

[0044] Figure 2 A schematic diagram of a process of abnormality detection provided by an embodiment of the present disclosure is shown.

[0045] Figure 3 A flowchart of a method for training a feature extraction model according to an embodiment of the present disclosure is shown.

[0046] Figure 4 A structural block diagram of a medical data anomaly detection device provided by an embodiment of the present disclosure is shown.

[0047] Figure 5 A structural block diagram of a feature extraction model training device provided by an embodiment of the present disclosure is shown.

[0048] Figure 6 A structural block diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0049] Figure 7 A schematic diagram showing the structure of a computer system suitable for implementing the method of the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0051] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0052] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] Figure 1 FIG. 1 is a flowchart of a method for detecting abnormalities in medical data provided by an embodiment of the present disclosure. Figure 1 As shown, the medical data anomaly detection method includes the following steps S101-S105:

[0054] In step S101, medical data to be detected is obtained, wherein the medical data to be detected includes target text data and target image data;

[0055] In step S102, a pre-trained first feature extractor is used to perform feature extraction on the target text data to obtain a target text vector feature;

[0056] In step S103, a pre-trained second feature extractor is used to extract features from the target image data to obtain target image features;

[0057] In step S104, the similarity between the target text vector feature and the target image feature is calculated;

[0058] In step S105 , if the similarity is less than or equal to a predetermined threshold, the medical data to be detected is abnormal; if the similarity is greater than the predetermined threshold, the medical data to be detected is normal.

[0059] In a possible implementation, the medical data anomaly detection method is applicable to a computer, computing device, server, server cluster, or other device that can perform medical data anomaly detection.

[0060] In a possible implementation, the medical data disclosed in the present invention is multimodal data, mainly including text data and image data, wherein the text data usually includes information such as the patient's condition description, related reactions and symptoms, test results, and treatment plans, and the image data usually includes scanned images of the patient's relevant diseased parts obtained by electronic computer tomography (CT, Computed Tomography), magnetic resonance imaging (MRI, Magnetic Resonance Imaging) and other methods. The text data is usually written by the attending physician, recording in detail the patient's clinical manifestations, medical history, diagnosis and treatment recommendations; while the image data is taken and selected by the radiologist to provide accurate visual lesion information. Since the people who write the text data and take the image data are different, there may be a mismatch between the text data and the image data of the same patient. This mismatch may be due to the following reasons:

[0061] First, medical text data is extremely complex and diverse; for example, text data written by different doctors may have different styles and focuses, and the same disease may show different symptom descriptions and characteristics under the descriptions of different doctors; secondly, there may also be differences in the use of medical terms, and different doctors may choose different words and expressions when recording the same condition; in addition, medical texts may also contain typos, grammatical errors or ambiguous descriptions, which further increases the possibility of matching deviations. On the other hand, medical images have a large amount of data and are information-intensive. Interpreting these images requires professional knowledge and experience, so image interpretation is prone to deviations; these deviations also lead to mismatches between text data and image data. These complex factors make it difficult to effectively match and screen massive amounts of medical text and image data through manual means, thereby increasing the difficulty and uncertainty of subsequent diagnosis and treatment. The solution provided by the present disclosure can automatically match and screen text data and image data, screen out abnormally matched text data and image data, reduce the difficulty and uncertainty of subsequent diagnosis and treatment, further ensure patient medical safety, and reduce physician errors.

[0062] In one possible implementation, Figure 2 A schematic diagram of an abnormality detection process provided by an embodiment of the present disclosure is shown, such as Figure 2 As shown, the medical data x to be detected new Include target text data and target image data For the target text data You can use a pre-trained first feature extractor To extract features, we can get the target text vector features. For target image data You can use a pre-trained second feature extractor To extract features, get the target image features Among them, φ * is the pre-trained first feature extractor The set of model parameters in θ * Pre-trained second feature extractor The set of model parameters in .

[0063] In one possible implementation, the pre-trained first feature extractor and the second feature extractor It can be sent to the device for anomaly detection after training by other servers, or it can be pre-trained by the device for anomaly detection. The two trained feature extractors can make the similarity between the text vector features and image features corresponding to the matched text data and image data larger, and the similarity between the text vector features and image features corresponding to the unmatched text data and image data smaller. In this way, after using these two feature extractors to perform feature extraction respectively, such as Figure 2 As shown, the extracted target text vector features are calculated and target image features If the similarity S is less than or equal to a predetermined threshold value d, it indicates that the target text data and the target image data in the medical data to be detected do not match, and there is an abnormality in the target text data and / or the target image data in the medical data to be detected. If the similarity S is greater than the predetermined threshold value d, it indicates that the target text data and the target image data in the medical data to be detected match, and both the target text data and the target image data in the medical data to be detected are normal.

[0064] After acquiring the medical data to be detected, this embodiment uses a pre-trained first feature extractor to perform feature extraction on the target text data to obtain target text vector features, and uses a pre-trained second feature extractor to perform feature extraction on the image data to obtain target image features; calculates the similarity between the target text vector features and the target image features, and determines that the medical data to be detected is abnormal when the similarity is less than or equal to a predetermined threshold, and determines that the medical data to be detected is normal when the similarity is greater than the predetermined threshold. Since the two feature extractors can be trained to make the similarity between the text vector features and the image features corresponding to the matching text data and image data greater, and the similarity between the text vector features and the image features corresponding to the mismatching text data and image data less, it avoids normal medical data being misjudged as abnormal medical data due to the text data and image data being misjudged as mismatched, and accurately performs anomaly detection on multimodal medical data.

[0065] In a possible implementation, the method further includes:

[0066] Obtaining matching normal medical multimodal datasets and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical imaging data, the value range of i and j includes 1, 2..., n, where n is a positive integer;

[0067] A feature extraction model is obtained by training using a normal medical multimodal data set and an abnormal data set matching an abnormality. The feature extraction model includes a first feature extractor and a second feature extractor. The first feature extractor is used to extract the feature from Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

[0068] In this embodiment, the matching normal medical multimodal data refers to the medical text data and medical image data describing the same disease. The matching normal medical multimodal data can be annotated by a physician or an expert in the field. The set formed by n matching normal medical multimodal data is recorded as a medical multimodal data set. At the same time, for comparative learning, medical multimodal datasets can be used Select medical text data and medical imaging data describing different diseases to form an abnormal data set Since the values ​​of i and j are different, and is a pair of abnormal multimodal data describing different diseases, is the i-th medical text data, is the jth image data, and does not match.

[0069] In this embodiment, a medical multimodal dataset matching normal and the anomaly dataset matching the anomaly To train a feature extraction model, the feature extraction model includes a first feature extractor and a second feature extractor. When performing model training, the first feature extractor T φ from Extracting text vector features from Using the second feature extractor C θ from Extracting image features After feature extraction is completed, the normal medical multimodal dataset is calculated and matched middle Corresponding text vector features and imaging features The first similarity of, and the abnormal dataset matching the anomaly middle Corresponding text vector features and imaging features The second similarity of the first feature extractor T is constantly adjusted during training. φ The model parameters φ and the second feature extractor C θ The model parameters θ in the formula are used to maximize the first similarity and minimize the second similarity. After the model converges or meets certain termination conditions to complete the training, the trained parameters φ are obtained. * and θ * , that is, the first feature extractor trained and the second feature extractor

[0070] For example, the optimization objective of the feature extraction model can be expressed by the following formula:

[0071]

[0072] Among them, S i To match normal medical multimodal datasets middle Corresponding text vector features and imaging features The first similarity, S j Anomaly dataset for matching anomalies middle Corresponding text vector features and imaging features The optimization goal is to maximize the first similarity and minimize the second similarity.

[0073] In a possible implementation, the use of a normal matching medical multimodal dataset and an abnormal matching abnormal dataset to train a feature extraction model includes:

[0074] From matching normal medical multimodal datasets Sample m samples in

[0075] Using the first feature extractor T φ right Extract features to get Using the second feature extractor C θ right Extract features to get

[0076] The following formula is used to update the model parameter φ in the first feature extractor in the s+1th cycle: s+1 :

[0077]

[0078] The following formula is used to update the model parameter θ in the second feature extractor in the s+1th cycle: s+1 :

[0079]

[0080] The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor;

[0081] Among them, φ s Indicates the first feature extractor T in the sth cycle φ The parameters in θ s Indicates the second feature extractor C in the sth cycle θ The parameters in , η is the predetermined step size, represents the gradient with respect to φ, represents the gradient with respect to θ, S aa for and The similarity between ab for and The similarity between .

[0082] In this embodiment, in the first cycle, the initial feature parameter φ in the first feature extractor can be set first. s＝1 and the initial feature parameters θ in the second feature extractor s＝1 , from matching normal medical multimodal datasets Randomly sample m samples from the dataset and use the first feature extractor T φ right Extract features to get Using the second feature extractor C θ right Extract features to get For medical multimodal datasets from matched normal Corresponding features calculate and The similarity S between aa , for the abnormal data set Corresponding features calculate and The similarity S between ab , in order to maximize the similarity S aa , minimize the similarity S ab , the above update formulas for θ and φ can be used to update θ and φ. The update order of the two parameters is not specific. The parameter θ in the second feature extraction can be updated first. s＝0 The gradient ascent algorithm is used to update the parameter φ in the second feature extraction in the s+1=1th cycle. s+1＝1 , then, the parameter φ in the first feature extraction s+1＝1 The gradient ascent algorithm is used to update the parameter θ in the second feature extractor in the s+1=1th cycle. s+1＝1 ; Or you can first extract the parameter φ in the first feature s＝0 The gradient ascent algorithm is used to update the parameter θ in the second feature extraction in the s+1=1th cycle. s+1＝1 , then, the parameter θ in the second feature extraction s+1＝1 The gradient ascent algorithm is used to update the parameter φ in the second feature extractor at the s+1=1th cycle. s+1＝1 After the update is completed, a second cycle is performed to match the normal medical multimodal dataset In the training, m samples are randomly sampled again and the above update formulas of θ and φ are used to update θ and φ again. This cycle is repeated many times until the model converges, that is, the changes of θ and φ are small and meet the predetermined conditions. The training is completed and the trained first feature extractor is obtained. and the second feature extractor It should be noted here that each time the m samples are sampled randomly, the m samples sampled at different times may be the same or different.

[0083] In a possible implementation, the first feature extractor includes a language model that converts word segments in text data into word vectors, and the second feature extractor includes a convolutional neural network.

[0084] In this implementation, the original text data cannot be directly analyzed and processed. First, it needs to be segmented to convert a medical text into a sequence of multiple words. For example, a preset segmenter can be used to convert the text data into a sequence of words. is converted into Where k is the number of words that the word segmenter divides the text data into.

[0085] In this embodiment, the language model can be a pre-trained language model that can be fine-tuned (such as a Bert (Bidirectional Encoder Representations from Transformers) language model), which can be used to convert word vectors into word vectors. Among them G φ (·) represents a language model with parameter φ. After obtaining the word vector, multiple methods such as (sum, average, etc.) can be used to obtain the final text vector features. For example, when the word vector average method is used to obtain the text vector features, the first feature extractor T φ From a text data The text vector features extracted from It can be expressed by the following formula:

[0086]

[0087] In this embodiment, the second feature extractor may be a convolutional neural network (CNN).

[0088] In other possible implementations, the first feature extractor and the second feature extractor may also be other types of neural networks, which are not limited here.

[0089] In a possible implementation, the similarity described in the present disclosure may be calculated using a Gaussian kernel function.

[0090] In this embodiment, a Gaussian kernel function with a bandwidth of r can be used to calculate the similarity. as well as The similarity S between aa It can be calculated using the following formula: For the above and The similarity S between ab , can be calculated using the following formula: For the above target text vector features and the target image features The similarity S between them can be calculated using the following formula Assuming the preset threshold is d, then:

[0091]

[0092] In other implementations, the similarity may also be calculated using other kernel functions or similarity measurement functions such as cosine similarity, which is not limited here.

[0093] In a possible implementation manner, the predetermined threshold is S ii The minimum value of S ii is the i-th pair of normally matched multimodal data and The similarity between the text vector features and the image features extracted by the trained first feature extractor and the second feature extractor is used.

[0094] In this embodiment, for a medical multimodal dataset You can use the trained first feature extractor and the second feature extractor Respectively and Perform feature extraction and calculate the similarity between the extracted text vector features and image features The value range of i is 1, ..., n, then the predetermined threshold d = min (S 11 ,S 22 ,...,S nn ).

[0095] The present disclosure also provides a method for training a feature extraction model. Figure 3 FIG. 1 is a flow chart showing a method for training a feature extraction model according to an embodiment of the present disclosure. Figure 3 As shown, the training method of the feature extraction model includes the following steps S301-S302:

[0096] In step S301, a medical multimodal data set matching a normal condition and an abnormal data set matching an abnormal condition are obtained;

[0097] Among them, medical multimodal dataset Anomaly dataset matching anomalies and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer;

[0098] In step S302, a feature extraction model is trained using a normal medical multimodal dataset and an abnormal dataset that matches an abnormality;

[0099] The feature extraction model includes a first feature extractor and a second feature extractor, wherein the first feature extractor is used to extract Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

[0100] In a possible implementation, the method for training the feature extraction model is applicable to computers, computing devices, servers, server clusters and other devices that can train feature extraction models.

[0101] In a possible implementation, the matched normal medical multimodal data refers to the medical text data and medical image data describing the same disease. The matched normal medical multimodal data can be annotated by a physician or an expert in the field. The set of n matched normal medical multimodal data is recorded as a medical multimodal data set. At the same time, for comparative learning, medical multimodal datasets can be used Select medical text data and medical imaging data describing different diseases to form an abnormal data set Since the values ​​of i and j are different, and is a pair of abnormal multimodal data describing different diseases, is the i-th medical text data, is the jth image data, and does not match.

[0102] In this embodiment, a medical multimodal dataset matching normal and the anomaly dataset matching the anomaly To train a feature extraction model, the feature extraction model includes a first feature extractor and a second feature extractor. When performing model training, the first feature extractor T φ from Extracting text vector features from Using the second feature extractor C θ from Extracting image features After feature extraction is completed, the normal medical multimodal dataset is calculated and matched middle Corresponding text vector features and imaging features The first similarity of, and the abnormal dataset matching the anomaly middle Corresponding text vector features and imaging features The second similarity of the first feature extractor T is constantly adjusted during training. φ The model parameters φ and the second feature extractor C θ The model parameters θ in the formula are used to maximize the first similarity and minimize the second similarity. After the model converges or meets certain termination conditions to complete the training, the trained parameters φ are obtained. * and θ * , that is, the first feature extractor trained and the second feature extractor

[0103] In a possible implementation, the use of a normal matching medical multimodal dataset and an abnormal matching abnormal dataset to train a feature extraction model includes:

[0104] From matching normal medical multimodal datasets Sample m samples in

[0105] Using the first feature extractor T φ right Extract features to get Using the second feature extractor C θ right Extract features to get

[0106] The following formula is used to update the model parameter φ in the first feature extractor in the s+1th cycle: s+1 :

[0107]

[0108] The following formula is used to update the model parameter θ in the second feature extractor in the s+1th cycle: s+1 :

[0109]

[0110] The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor;

[0111] Among them, φ s Indicates the first feature extractor T in the sth cycle φ The parameters in θ s Indicates the second feature extractor C in the sth cycle θ The parameters in , η is the predetermined step size, represents the gradient with respect to φ, represents the gradient with respect to θ, S aa for and The similarity between ab for and The similarity between .

[0112] In this embodiment, in the first cycle, the initial feature parameter φ in the first feature extractor can be set first. s＝1 and the initial feature parameters θ in the second feature extractor s＝1 , from matching normal medical multimodal datasets Randomly sample m samples from the dataset and use the first feature extractor T φ right Extract features to get Using the second feature extractor C θ right Extract features to get For medical multimodal datasets from matched normal Corresponding features calculate and The similarity S between aa , for the abnormal data set Corresponding features calculate and The similarity S between ab , in order to maximize the similarity S aa , minimize the similarity S ab , the above update formulas for θ and φ can be used to update θ and φ. The update order of the two parameters is not specific. The parameter θ in the second feature extraction can be updated first. s＝0 The gradient ascent algorithm is used to update the parameter φ in the second feature extraction in the s+1=1th cycle. s+1＝1 , then, the parameter φ in the first feature extraction s+1＝1 The gradient ascent algorithm is used to update the parameter θ in the second feature extractor in the s+1=1th cycle.s+1＝1 ; Or you can first extract the parameter φ in the first feature s＝0 The gradient ascent algorithm is used to update the parameter θ in the second feature extraction in the s+1=1th cycle. s+1＝1 , then, the parameter θ in the second feature extraction s+1＝1 The gradient ascent algorithm is used to update the parameter φ in the second feature extractor at the s+1=1th cycle. s+1＝1 After the update is completed, a second cycle is performed to match the normal medical multimodal dataset In the training, m samples are randomly sampled again and the above update formulas of θ and φ are used to update θ and φ again. This cycle is repeated many times until the model converges, that is, the changes of θ and φ are small and meet the predetermined conditions. The training is completed and the trained first feature extractor is obtained. and the second feature extractor It should be noted here that each time the m samples are sampled randomly, the m samples sampled at different times may be the same or different.

[0113] In a possible implementation, the first feature extractor includes a language model that converts word segments in text data into word vectors, and the second feature extractor includes a convolutional neural network.

[0114] In this implementation, the original text data cannot be directly analyzed and processed. First, it needs to be segmented to convert a medical text into a sequence of multiple words. For example, a preset segmenter can be used to convert the text data into a sequence of words. is converted into Where k is the number of words that the word segmenter divides the text data into.

[0115] In this embodiment, the language model can be a pre-trained language model that can be fine-tuned (such as a Bert (Bidirectional Encoder Representations from Transformers) language model), which can be used to convert word vectors into word vectors. Among them G φ (·) represents a language model with parameter φ. After obtaining the word vector, multiple methods such as (sum, average, etc.) can be used to obtain the final text vector features. For example, when the word vector average method is used to obtain the text vector features, the first feature extractor T φ From a text data The text vector features extracted from It can be expressed by the following formula:

[0116]

[0117] In this embodiment, the second feature extractor may be a convolutional neural network (CNN).

[0118] In other possible implementations, the first feature extractor and the second feature extractor may also be other types of neural networks, which are not limited here.

[0119] In a possible implementation, the similarity described in the present disclosure may be calculated using a Gaussian kernel function.

[0120] In this embodiment, a Gaussian kernel function with a bandwidth of r can be used to calculate the similarity. as well as The similarity S between aa It can be calculated using the following formula: For the above and The similarity S between ab , can be calculated using the following formula: For the above target text vector features and the target image features The similarity S between them can be calculated using the following formula Assuming the preset threshold is d, then:

[0121]

[0122] In other implementations, the similarity may also be calculated using other kernel functions or similarity measurement functions such as cosine similarity, which is not limited here.

[0123] The present disclosure also provides a device for detecting abnormality in medical data. Figure 4 The structure block diagram of a medical data anomaly detection device provided by an embodiment of the present disclosure is shown. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the medical data anomaly detection device includes:

[0124] The data acquisition module 401 is configured to acquire medical data to be detected, wherein the medical data to be detected includes target text data and target image data;

[0125] The feature extraction module 402 is configured to use a pre-trained first feature extractor to perform feature extraction on the target text data to obtain target text vector features; and use a pre-trained second feature extractor to perform feature extraction on the image data to obtain target image features;

[0126] A calculation module 403 is configured to calculate the similarity between the target text vector feature and the target image feature;

[0127] The detection module 404 is configured to determine that the medical data to be detected is abnormal if the similarity is less than or equal to a predetermined threshold, and that the medical data to be detected is normal if the similarity is greater than the predetermined threshold.

[0128] In a possible implementation, the device further includes:

[0129] The model training module is configured to: obtain a medical multimodal dataset that matches normal and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth medical image data different from the ith medical image data, the value range of i and j includes 1, 2..., n, and n is a positive integer; using a normal medical multimodal data set and an abnormal data set matching an abnormality, a feature extraction model is trained, and the feature extraction model includes a first feature extractor and a second feature extractor, and the first feature extractor is used to extract the first feature from Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to minimize and The similarity between the corresponding text vector features and image features is maximized and The similarity between the corresponding text vector features and image features.

[0130] In a possible implementation, the model training module uses a normal medical multimodal data set and an abnormal data set that matches an abnormality, and the part of the feature extraction model obtained by training is configured as follows:

[0131] From matching normal medical multimodal datasets Sample m samples in

[0132] Using the first feature extractor Tφ right Extract features to get Using the second feature extractor C θ right Extract features to get

[0133] The following formula is used to update the model parameter φ in the first feature extractor in the s+1th cycle: s+1 :

[0134]

[0135] The following formula is used to update the model parameter θ in the second feature extractor in the s+1th cycle: s+1 :

[0136]

[0137] The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor;

[0138] Among them, φ s Indicates the first feature extractor T in the sth cycle φ The parameters in θ s Indicates the second feature extractor C in the sth cycle θ The parameters in , η is the predetermined step size, represents the gradient with respect to φ, represents the gradient with respect to θ, S aa for and The similarity between ab for and The similarity between .

[0139] In a possible implementation, the first feature extractor includes a language model that converts word segments in text data into word vectors, and the second feature extractor includes a convolutional neural network.

[0140] In a possible implementation, the similarity is calculated using a Gaussian kernel function.

[0141] In a possible implementation manner, the predetermined threshold is S ii The minimum value of S ii is the i-th pair of normally matched multimodal data and The similarity between the text vector features and the image features extracted by the trained first feature extractor and the second feature extractor is used.

[0142] The present disclosure also provides a training device for a feature extraction model. Figure 5 The structural block diagram of a feature extraction model training device provided by an embodiment of the present disclosure is shown. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 5 As shown, the training device of the feature extraction model includes:

[0143] The data set acquisition module 501 is configured to acquire a medical multimodal data set that matches a normal and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer;

[0144] The training module 502 is configured to use the normal medical multimodal data set and the abnormal data set to train a feature extraction model, wherein the feature extraction model includes a first feature extractor and a second feature extractor. The first feature extractor is used to extract the feature from the normal medical multimodal data set and the abnormal data set. Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

[0145] In a possible implementation, the training module is configured as follows:

[0146] From matching normal medical multimodal datasets Sample m samples in

[0147] Using the first feature extractor T φ right Extract features to get Using the second feature extractor C θ right Extract features to get

[0148] The following formula is used to update the model parameter φ in the first feature extractor in the s+1th cycle: s+1:

[0149]

[0150] The following formula is used to update the model parameter θ in the second feature extractor in the s+1th cycle: s+1 :

[0151]

[0152] The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor;

[0153]

[0154] The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor;

[0155] In a possible implementation, the first feature extractor includes a language model that converts word segments in text data into word vectors, and the second feature extractor includes a convolutional neural network.

[0156] In a possible implementation, the similarity is calculated using a Gaussian kernel function.

[0157] The technical terms and technical features mentioned in the implementation manner of the present device are the same as or similar to those mentioned in the implementation manner of the above method. For the explanation and description of the technical terms and technical features involved in the present device, reference may be made to the explanation and description of the implementation manner of the above method, and they will not be repeated here.

[0158] The present disclosure also discloses an electronic device, Figure 6 A structural block diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0159] like Figure 6 As shown, the electronic device 600 includes a memory 601 and a processor 602, wherein the memory 601 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 602 to implement the method according to an embodiment of the present disclosure.

[0160] Figure 7 A schematic diagram showing the structure of a computer system suitable for implementing the method of the embodiment of the present disclosure is shown.

[0161] like Figure 7As shown, the computer system 700 includes a processing unit 701, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the computer system 700 are also stored. The processing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0162] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed. Among them, the processing unit 701 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0163] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes computer instructions, and the computer instructions are executed by a processor to implement the method steps described above. In such an embodiment, the computer program product can be downloaded and installed from a network through the communication part 709, and / or installed from a removable medium 711.

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or programmable hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0166] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0167] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A method for detecting anomalies in medical data, characterized in that: The method comprises: Acquiring medical data to be detected, wherein the medical data to be detected includes target text data and target image data; Using a pre-trained first feature extractor to extract features from the target text data to obtain target text vector features; Using a pre-trained second feature extractor to extract features from the target image data to obtain target image features; Calculating the similarity between the target text vector feature and the target image feature; If the similarity is less than or equal to a predetermined threshold, the medical data to be detected is abnormal; if the similarity is greater than the predetermined threshold, the medical data to be detected is normal.

2. The anomaly detection method according to claim 1, characterized in that: The method further comprises: Obtaining matching normal medical multimodal datasets and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer; A feature extraction model is obtained by training using a normal medical multimodal data set and an abnormal data set matching an abnormality. The feature extraction model includes a first feature extractor and a second feature extractor. The first feature extractor is used to extract the feature from Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize x i 1 and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

3. The anomaly detection method according to claim 2, characterized in that: The method uses a normal medical multimodal data set and an abnormal data set to train a feature extraction model, including: From matching normal medical multimodal datasets Sample m samples in Using the first feature extractor T φ right Extract features to get Use the second feature extractor C θ right Extract features to get The following formula is used to update the model parameter φ in the first feature extractor in the s+1th cycle: s+1 : The following formula is used to update the model parameter θ in the second feature extractor in the s+1th cycle: s+1 : The above steps are repeated until the feature extraction model converges and the training is completed, thereby obtaining the trained first feature extractor and the second feature extractor; Among them, φ s Indicates the first feature extractor T in the sth cycle φ The parameters in θ s Indicates the second feature extractor C in the sth cycle θ The parameters in , η is the predetermined step size, represents the gradient with respect to φ, represents the gradient with respect to θ, S aa for and The similarity between ab for and The similarity between .

4. The anomaly detection method according to claim 1, characterized in that: The first feature extractor includes a language model that converts word segments in text data into word vectors, and the second feature extractor includes a convolutional neural network.

5. The anomaly detection method according to any one of claims 1 to 3, characterized in that: The similarity is calculated using a Gaussian kernel function.

6. The anomaly detection method according to claim 2, characterized in that: The predetermined threshold is S ii The minimum value of S ii is the i-th pair of normally matched multimodal data and The similarity between the text vector features and the image features extracted by the trained first feature extractor and the second feature extractor is used.

7. A method for training a feature extraction model, characterized in that: include: Obtaining matching normal medical multimodal datasets and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer; A feature extraction model is obtained by training using a normal medical multimodal data set and an abnormal data set matching an abnormality. The feature extraction model includes a first feature extractor and a second feature extractor. The first feature extractor is used to extract the feature from Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

8. A device for detecting abnormality in medical data, characterized in that: The device comprises: A data acquisition module is configured to acquire medical data to be detected, wherein the medical data to be detected includes target text data and target image data; The feature extraction module is configured to use a pre-trained first feature extractor to perform feature extraction on the target text data to obtain target text vector features; and use a pre-trained second feature extractor to perform feature extraction on the image data to obtain target image features; A calculation module, configured to calculate the similarity between the target text vector feature and the target image feature; The detection module is configured to determine that if the similarity is less than or equal to a predetermined threshold, the medical data to be detected is abnormal, and if the similarity is greater than the predetermined threshold, the medical data to be detected is normal.

9. A training device for a feature extraction model, characterized in that: include: A dataset acquisition module is configured to acquire a matching normal medical multimodal dataset and the anomaly dataset matching the anomaly in, and is the i-th pair of normally matched multimodal data, is the i-th medical text data, is the i-th medical imaging data, is the jth piece of medical image data that is different from the ith piece of medical image data, and the value ranges of i and j include 1, 2, ..., n, where n is a positive integer; The training module is configured to use a normal medical multimodal data set and an abnormal data set that matches an abnormality to train a feature extraction model, wherein the feature extraction model includes a first feature extractor and a second feature extractor, wherein the first feature extractor is used to extract the feature from the normal medical multimodal data set and the abnormal data set that matches the abnormality. Extracting text vector features, the second feature extractor is used to extract text vector features from The image features are extracted from the image. The optimization goal of the feature extraction model is to maximize and Minimize the similarity between the corresponding text vector features and image features and The similarity between the corresponding text vector features and image features.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

11. A readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images