Method and system for identifying acute and critical diseases based on multi-modal data
Through the multimodal data, combined with text, images, sound and physiological signals and other data, machine learning models are used to identify critical and critical illnesses, which solves the accuracy and timeliness of traditional emergency diagnosis relying on a single data. A more comprehensive and accurate condition assessment was achieved.
Patent Information
- Application Number
- CN202411341163.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional emergency diagnosis relies on a single type of data, resulting in the diagnosis results that may be inaccurate or timely to be able to fully evaluate the patient's condition.
Using the emergency and critical illness recognition method based on multimodal data, by collecting and cleaning various data types such as text, images, sounds and physiological signals, using machine learning models to extract features and generate judgment models, optimize model parameters, and identify potential emergency Critical cases and generate early warning signals to assist doctors in making decisions.
It improves the comprehensiveness and accuracy of the diagnosis, can identify critical and critical illness more in a timely manner, and improves the quality of life of patients.
Smart Images

Figure CN118969312A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of digital medical technology, and in particular to a method and system for identifying critical and severe diseases based on multimodal data. Background Art
[0002] Traditional emergency diagnosis often relies on a single type of data, such as a doctor's physical examination, blood tests, and routine imaging examinations. However, this approach has limitations. For example, for some diseases, a single type of data may not be enough to provide comprehensive information, resulting in a diagnosis that may not be accurate or timely.
[0003] Therefore, how to provide a method for identifying critically ill patients based on a critical care knowledge base and multimodal data such as 120 emergency telephone online voice and video is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention
[0004] To achieve the purpose of the present invention, the present application provides a method for identifying critical and severe diseases based on multimodal data, comprising:
[0005] Step S1: Collecting multimodal historical medical record data of the patient, and cleaning and standardizing the multimodal historical medical record data; the multimodal historical medical record data includes multiple data types including text, image, sound and physiological signal;
[0006] Step S2: extracting lesion features and key diagnostic information from the multimodal historical medical record data to form a feature set, and fusing the feature set using a machine learning model to generate a judgment model;
[0007] Step S3: using the multimodal historical medical record data to train the judgment model, optimize model parameters, and evaluate the performance of the judgment model;
[0008] Step S4: Input the patient's real-time medical data into the judgment model to identify potential critical cases and generate early warning signals, notify medical personnel of the identification results, and provide reference knowledge by searching the preset knowledge base to assist doctors in making decisions.
[0009] In some specific embodiments, in step S1, medical record text data, medical image data and sensor data in the hospital information system are collected; the cleaning and standardization processing includes noise removal, format standardization, and data labeling.
[0010] In some specific embodiments, the cleaning and standardization process includes:
[0011] Use the data cleaning tool Python's Pandas library to identify and process outliers, missing values, and noise in the data;
[0012] Standardize different data types, convert the medical image data into a standard image format, extract the word segmentation and stems of the medical record text data through natural language processing technology, and extract the frequency domain and time domain features of the sensor data through signal processing technology and perform normalization processing.
[0013] In some specific embodiments, in step S2, a feature selection algorithm is used to extract and mark the most representative and relevant features in the feature set, and then the marked features are used to train the judgment model, and the parameters of the judgment model are adjusted and optimized through the back propagation algorithm and the stochastic gradient descent algorithm.
[0014] In some specific embodiments, step S3 also includes: pre-storing critical and severe disease text and critical and severe disease pictures, and using a BERT pre-trained model to extract descriptive features of the critical and severe disease text, and using a SwinTransformer model to extract image features of the critical and severe disease pictures.
[0015] In some specific embodiments, the BERT pre-training model is used to extract the descriptive features of the critical and severe text T = {t0, t1, ... t m-1}, the calculation formula is as follows:
[0016] T=W f (BERT(S tokenizer ))
[0017] Among them, m is the dimension of text features, t m-1 For each dimension of text features, S tokennizer is the text after word segmentation, W f is a linear transformation.
[0018] In some specific embodiments, the local features of the critical and severe pictures are extracted using the Swin Transformer model, and the image features I = {i0, i1, ... i n-1}, the calculation formula is as follows:
[0019] I=W f (ST(Image))
[0020] Among them, n represents the number of grids used to divide the feature map in the Swin Transformer model, i n-1 is a local feature, ST represents a backbone network based on the Swin Transformer model, Image is the critical illness picture, and W f is a linear transformation.
[0021] In some specific embodiments, step S3 also includes using a double-tower structure to compare and learn the historical medical record data and the patient's real-time condition data, the historical medical record data includes historical text features and historical image features, and the real-time condition data includes real-time text features and real-time image features, and cross-calculates the similarity of features between each modality according to the following formula:
[0022] S(I,T)=g v (V cls ) T g′ w (W c ' ls )
[0023] S(T,I)=g w (W cls ) T g v ′(V c ' ls )
[0024] Among them, S(I,T) represents the similarity score between historical image features and real-time text features, S(T,I) represents the similarity score between historical text features and real-time image features, and g v (V cls ) and g w (W cls ) represent the linear transformation of image features and text features respectively, g v ′(V c ' ls ) and g′ w (W c ' ls ) are respectively converted by momentum encoder to g v (V cls ) and g w (W cls ) is used to encode the image features and text features.
[0025] In some specific embodiments, step S4 further includes setting a predetermined threshold to determine whether the case is an emergency or critical illness, and controlling the sensitivity and specificity of the recognition result by adjusting the threshold;
[0026] The preprocessed data is input into the trained judgment model to obtain real-time recognition results, and whether it is an emergency or critical case is determined based on the probability or classification result output by the model. The recognition results are fed back to medical staff in real time, and medical staff are reminded to deal with emergency and critical cases in a timely manner through alarm systems, displays or mobile applications.
[0027] To achieve the same invention purpose, the present application also provides an emergency and critical illness identification system based on multimodal data, including:
[0028] Data collection module: used to collect multimodal historical medical record data of patients, and to clean and standardize the multimodal historical medical record data; the multimodal historical medical record data includes multiple data types such as text, image, sound and physiological signal;
[0029] Model generation module: used to extract lesion features and key diagnostic information from the multimodal historical medical record data to form a feature set, fuse the feature set using a machine learning model, and generate a judgment model;
[0030] Model optimization module: used to train the judgment model using the multimodal historical medical record data, optimize model parameters, and evaluate the performance of the judgment model;
[0031] Result generation module: used to input the patient's real-time medical data into the judgment model to identify potential critical and severe cases and generate early warning signals, notify medical personnel of the identification results, and provide reference knowledge by searching the preset knowledge base to assist doctors in making decisions.
[0032] Beneficial effects of the above technical solution:
[0033] The critical and severe disease identification technology based on multimodal data provided by this application can use multiple data sources at the same time, thereby improving the comprehensiveness and accuracy of diagnosis. First, data collection and preprocessing are required, including collecting data from various medical devices and systems, and cleaning and standardizing the data to ensure the quality and consistency of the data. Through feature extraction and data fusion models, data of different modalities are integrated together to form a unified feature space. The model is trained and optimized using historical medical record data, and the patient's real-time data is input into the model to identify potential critical and severe cases and generate corresponding early warning signals. The early warning system is used to notify medical personnel and provide corresponding diagnosis and treatment recommendations to support doctors in making decisions. This application can more comprehensively evaluate the patient's condition, improve the accuracy and timeliness of diagnosis, and better treat patients and improve their quality of life by comprehensively utilizing different types of data. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 A flowchart of a method for identifying critical and severe diseases based on multimodal data provided by an embodiment of the present invention;
[0036] Figure 2 A schematic diagram of the structure of a critical illness identification system based on multimodal data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0038] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0039] Embodiment 1
[0040] An embodiment of the present invention provides a method for identifying critical and severe diseases based on multimodal data. Figure 1 As shown, including:
[0041] Step S1: Collecting multimodal historical medical record data of the patient, and cleaning and standardizing the multimodal historical medical record data; the multimodal historical medical record data includes multiple data types including text, image, sound and physiological signal;
[0042] Step S2: extracting lesion features and key diagnostic information from the multimodal historical medical record data to form a feature set, and fusing the feature set using a machine learning model to generate a judgment model;
[0043] Step S3: using the multimodal historical medical record data to train the judgment model, optimize model parameters, and evaluate the performance of the judgment model;
[0044] Step S4: Input the patient's real-time medical data into the judgment model to identify potential critical cases and generate early warning signals, notify medical personnel of the identification results, and provide reference knowledge by searching the preset knowledge base to assist doctors in making decisions.
[0045] In a specific embodiment of the present invention, in step S1, medical record text data, medical image data and sensor data in the hospital information system are collected; the cleaning and standardization processing includes noise removal, format standardization, and data labeling.
[0046] In a specific embodiment of the present invention, the cleaning and standardization process comprises:
[0047] Use the data cleaning tool Python's Pandas library to identify and process outliers, missing values, and noise in the data;
[0048] Standardize different data types, convert the medical image data into a standard image format, extract the word segmentation and stems of the medical record text data through natural language processing technology, and extract the frequency domain and time domain features of the sensor data through signal processing technology and perform normalization processing.
[0049] In a specific embodiment of the present invention, in step S2, a feature selection algorithm is used to extract the most representative and relevant features in the feature set and mark them, and then the marked features are used to train the judgment model, and the parameters of the judgment model are adjusted and optimized through the back propagation algorithm and the stochastic gradient descent algorithm.
[0050] In a specific embodiment of the present invention, step S3 also includes: pre-storing critical and severe disease text and critical and severe disease pictures, and using the BERT pre-trained model to extract the descriptive features of the critical and severe disease text, and using the SwinTransformer model to extract the image features of the critical and severe disease pictures.
[0051] In a specific embodiment of the present invention, the description features of the critical and severe text are extracted using the BERT pre-training model T = {t0, t1, ... t m-1}, the calculation formula is as follows:
[0052] T=W f (BERT(S tokenizer ))
[0053] Among them, m is the dimension of text features, t m-1 For each dimension of text features, S tokennizer is the text after word segmentation, W f is a linear transformation.
[0054] In a specific embodiment of the present invention, the local features of the critical and severe pictures are extracted using the Swin Transformer model, and the image features I = {i0, i1, ... i n-1}, the calculation formula is as follows:
[0055] I=W f (ST(Image))
[0056] Among them, n represents the number of grids used to divide the feature map in the Swin Transformer model, i n-1is a local feature, ST represents a backbone network based on the Swin Transformer model, Image is the critical illness picture, and W f is a linear transformation.
[0057] In a specific embodiment of the present invention, step S3 also includes using a dual-tower structure to compare and learn the historical medical record data and the patient's real-time condition data, the historical medical record data includes historical text features and historical image features, and the real-time condition data includes real-time text features and real-time image features, and cross-calculates the similarity of features between each modality according to the following formula:
[0058] S(I,T)=g v (V cls ) T g′ w (W c ' ls )
[0059] S(T,I)=g w (W cls ) T g v ′(V c ' ls )
[0060] Among them, S(I,T) represents the similarity score between historical image features and real-time text features, S(T,I) represents the similarity score between historical text features and real-time image features, and g v (V cls ) and g w (W cls ) represent the linear transformation of image features and text features respectively, g v ′(V c ' ls ) and g′ w (W c ' ls ) are respectively converted by momentum encoder to g v (V cls ) and g w (W cls ) is used to encode the image features and text features.
[0061] Perform Softmax operations on S(I,T) and S(T,I) respectively to obtain the similarity between historical image features and real-time text features. The similarity between historical text features and real-time image features
[0062]
[0063] Use the image-text comparison loss function L itc Optimize the feature alignment module, the loss function is as follows:
[0064]
[0065] Among them, E (I,T)~D Indicates the alignment degree between text features and image features I; H(y i2t (I),p t2i (I)) represents the similarity between the image feature I and the text feature after transformation; H(y t2i (T),p t2ii (T)) represents the similarity between the text feature T and the image feature after transformation.
[0066] The category feature and position feature are added to the text feature T and the image feature I respectively to obtain i′ and t′ respectively. The calculation formula is as follows:
[0067] t'=[t0M t ;t1M t ;...;t m-1 M t ]+M tpos +M ttype
[0068] i'=[i0M i ;i1M;...;i n-1 M]+M pos +M type
[0069] Among them, M pos and M pos Respectively represent the position encoding of this article and the position encoding of the image, M type and M type Represent text category encoding and image category encoding respectively.
[0070] The aligned text feature T and image feature I are input into the SwinTransformer model containing the cross attention module, and the image feature I and text feature T are merged to obtain the merged feature Z0, which is calculated as follows:
[0071] Z0=[t';i']
[0072] Input Z0 into the fusion module TFM based on the Swin Transformer model to obtain the fused image-text feature Z L , and its calculation formula is as follows:
[0073] z L =TFM(z0)
[0074] The feature Z L Input into the classifier module to obtain the category C of critical illness, and the calculation formula is as follows:
[0075] C=Softmax(Linear(Z L ))
[0076] Among them, Linear represents the linear layer.
[0077] In a specific embodiment of the present invention, step S4 further includes setting a predetermined threshold to determine whether the case is an emergency or critical illness, and controlling the sensitivity and specificity of the recognition result by adjusting the threshold;
[0078] The preprocessed data is input into the trained judgment model to obtain real-time recognition results, and whether it is an emergency or critical case is determined based on the probability or classification result output by the model. The recognition results are fed back to medical staff in real time, and medical staff are reminded to deal with emergency and critical cases in a timely manner through alarm systems, displays or mobile applications.
[0079] Specifically, in the real-time monitoring stage, an abnormal detection mechanism is set up to monitor whether the patient data exceeds the normal range. When critical symptoms are detected, the alarm system is triggered to notify medical staff in a timely manner. Different priority alarms can be set according to the patient's condition and severity so that medical staff can respond quickly. Based on the identification results and the patient's personal information, corresponding diagnostic suggestions are provided to assist medical staff in making further diagnosis and treatment decisions. Real-time monitoring functions are provided, allowing medical staff to view the patient's physiological parameters and diagnostic results at any time, so as to adjust the treatment strategy in a timely manner.
[0080] The critical and severe disease identification technology based on multimodal data provided by this application can use multiple data sources at the same time, thereby improving the comprehensiveness and accuracy of diagnosis. First, data collection and preprocessing are required, including collecting data from various medical devices and systems, and cleaning and standardizing the data to ensure the quality and consistency of the data. Through feature extraction and data fusion models, data of different modalities are integrated together to form a unified feature space. The model is trained and optimized using historical medical record data, and the patient's real-time data is input into the model to identify potential critical and severe cases and generate corresponding early warning signals. The early warning system is used to notify medical personnel and provide corresponding diagnosis and treatment recommendations to support doctors in making decisions. This application can more comprehensively evaluate the patient's condition, improve the accuracy and timeliness of diagnosis, and better treat patients and improve their quality of life by comprehensively utilizing different types of data.
[0081] Embodiment 2
[0082] An embodiment of the present invention provides an emergency and critical illness identification system based on multimodal data, referring to Figure 2 As shown, including:
[0083] Data collection module 10: used to collect multimodal historical medical record data of patients, and clean and standardize the multimodal historical medical record data; the multimodal historical medical record data includes multiple data types such as text, image, sound and physiological signal;
[0084] Model generation module 20: used to extract lesion features and key diagnostic information from the multimodal historical medical record data to form a feature set, fuse the feature set using a machine learning model, and generate a judgment model;
[0085] Model optimization module 30: used to train the judgment model using the multimodal historical medical record data, optimize model parameters, and evaluate the performance of the judgment model;
[0086] Result generation module 40: used to input the patient's real-time medical data into the judgment model to identify potential critical and severe cases and generate early warning signals, notify medical personnel of the identification results, and provide reference knowledge by searching a preset knowledge base to assist doctors in making decisions.
[0087] In a specific embodiment of the present invention, data is collected from various data sources using appropriate interfaces or software tools and stored in an appropriate data warehouse to ensure the integrity and confidentiality of the data.
[0088] Specifically, in addition to the 120 Center's own database, it also includes:
[0089] Medical databases: Medical databases such as PubMed, MEDLINE, and EMBASE provide a large amount of medical literature and research abstracts;
[0090] Clinical trial registration databases: Clinical trial registration databases such as ClinicalTrials.gov and the China Clinical Trial Registry collect clinical trial information worldwide;
[0091] 120 Centers’ Electronic Medical Records and Clinical Databases: Many medical institutions use electronic medical record systems and clinical databases to record and manage patients’ medical information.
[0092] In a specific embodiment of the present invention, the most representative and relevant features are selected from the extracted feature set using feature selection algorithms, such as variance threshold, correlation coefficient, etc. The model is trained using labeled training data, and optimization methods such as back propagation algorithm and stochastic gradient descent can be used to adjust model parameters to capture the relationship between data as accurately as possible.
[0093] In a specific embodiment of the present invention, the BERT pre-trained model is used to extract the features of the critical and severe text descriptions uploaded by doctors, and the Swin Transformer model is used to extract the features of the critical and severe patient images. The performance of the model is evaluated on a validation set or a cross-validation set, and evaluation indicators such as accuracy, recall, and F1 value are used to evaluate the effect of the model.
[0094] The real-time data of patients is input into the model to identify potential critical and severe cases and generate early warning signals. The identification results are notified to medical staff through the early warning system. By searching the knowledge base and providing diagnosis and treatment suggestions, doctors are assisted in making decisions, including:
[0095] Critical illness identification: According to the application scenario and needs, set the appropriate threshold to determine whether the case is an emergency or critical illness. The sensitivity and specificity of the identification result can be controlled by adjusting the threshold. The pre-processed data is input into the trained model to obtain real-time identification results. According to the probability or classification results output by the model, it is determined whether it is an emergency or critical illness case. The identification results are fed back to medical staff in real time, and medical staff are reminded to deal with emergency and critical illness cases in a timely manner through alarm systems, displays or mobile applications.
[0096] Early warning and decision support: In the real-time monitoring stage, an abnormal detection mechanism is set up to monitor whether the patient data exceeds the normal range. When critical symptoms are detected, the alarm system is triggered to notify medical staff in time. Different priority alarms can be set according to the patient's condition and severity so that medical staff can respond quickly. According to the identification results and the patient's personal information, corresponding diagnostic suggestions are provided to assist medical staff in making further diagnosis and treatment decisions. Real-time monitoring functions are provided, allowing medical staff to view the patient's physiological parameters and diagnostic results at any time, so as to adjust the treatment strategy in time.
[0097] This system aims to identify critically ill patients based on the critical and emergency knowledge base and multimodal data such as online voice and video of 120 emergency calls. Online voice and video are two modal data with different feature space distributions. In order to make the two data represent common semantics, the emergency diagnosis and treatment decision system for critical and emergency diseases needs to align cross-modal data and map them to the same feature semantic space. In order to identify the corresponding critical and emergency diseases, the aligned multimodal semantic features are fused to enhance their relevance, and then the model classifier is used to increase the differences between different critical and emergency diseases to classify them. The identified critical and emergency diseases are based on historical medical records composed of critical and emergency knowledge graphs. The historical medical records and historical diagnosis and treatment plans similar to the critical and emergency diseases are obtained through the similarity matching retrieval algorithm. Specific first aid measures will be displayed in the form of video on the video surveillance on the ambulance to assist medical staff in remotely guiding callers to perform first aid.
[0098] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
[0099] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other. The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention. Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0100] The method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "one specific embodiment" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for identifying critical and severe diseases based on multimodal data, characterized in that: include: Step S1: collecting multimodal historical medical record data of the patient, and cleaning and standardizing the multimodal historical medical record data; The multimodal historical medical record data includes multiple data types including text, image, sound and physiological signal; Step S2: extracting lesion features and key diagnostic information from the multimodal historical medical record data to form a feature set, and fusing the feature set using a machine learning model to generate a judgment model; Step S3: using the multimodal historical medical record data to train the judgment model, optimize model parameters, and evaluate the performance of the judgment model; Step S4: Input the patient's real-time medical data into the judgment model to identify potential critical cases and generate early warning signals, notify medical personnel of the identification results, and provide reference knowledge by searching the preset knowledge base to assist doctors in making decisions.
2. The method for identifying critical and severe diseases based on multimodal data according to claim 1, characterized in that: Step S1 includes: collecting medical record text data, medical image data and sensor data in the hospital information system; the cleaning and standardization processing includes removing noise, standardizing the format and data labeling.
3. The method for identifying critical and severe diseases based on multimodal data according to claim 2, characterized in that: The cleaning and standardization process includes: Use the data cleaning tool Python's Pandas library to identify and process outliers, missing values, and noise in the data; Standardize different data types, convert the medical image data into a standard image format, extract the word segmentation and stems of the medical record text data through natural language processing technology, and extract the frequency domain and time domain features of the sensor data through signal processing technology and perform normalization processing.
4. The method for identifying critical and severe diseases based on multimodal data according to claim 1, characterized in that: In step S2, a feature selection algorithm is used to extract the most representative and relevant features in the feature set and mark them, and then the marked features are used to train the judgment model, and the parameters of the judgment model are adjusted and optimized through the back propagation algorithm and the stochastic gradient descent algorithm.
5. The method for identifying critical and severe diseases based on multimodal data according to claim 1, characterized in that: Step S3 also includes: pre-storing critical and severe disease text and critical and severe disease pictures, and using the BERT pre-trained model to extract the description features of the critical and severe disease text, and using the Swin Transformer model to extract the image features of the critical and severe disease pictures.
6. The method for identifying critical and severe diseases based on multimodal data according to claim 5, characterized in that: The BERT pre-training model is used to extract the descriptive features of the critical and severe text T = {t0, t1, ... t m-1 }, the calculation formula is as follows: T=W f (BERT(S tokenizer )) Among them, m is the dimension of text features, t m-1 For each dimension of text features, S tokennizer is the text after word segmentation, W f is a linear transformation.
7. The method for identifying critical and severe diseases based on multimodal data according to claim 1, characterized in that: The SwinTransformer model is used to extract the local features of the critical and severe pictures, and the image features I = {i0, i1, ... i n-1 }, the calculation formula is as follows: I=W f (ST(Image)) Among them, n represents the number of grids used to divide the feature map in the Swin Transformer model, i n-1 is the Embedding of local features, ST represents the backbone network based on the Swin Transformer model, Image is the critical illness picture, and W f is a linear transformation.
8. The method for identifying critical and severe diseases based on multimodal data according to claim 1, characterized in that: Step S3 also includes using a double-tower structure to compare and learn the historical medical record data and the patient's real-time condition data, wherein the historical medical record data includes historical text features and historical picture features, and the real-time condition data includes real-time text features and real-time picture features, and the similarity between the text features and the picture features is cross-calculated according to the following formula: S(I,T)=g v (V cls ) T g′ w (W c ′ ls ) S(T,I)=g w (W cls ) T g v ′(V c ′ ls ) Among them, S(I,T) represents the similarity score between historical image features and real-time text features, S(T,I) represents the similarity score between historical text features and real-time image features, and g v (V cls ) and g w (W cls ) represent the linear transformation of image features and text features respectively, g v ′(V c ' ls ) and g′ w (W c ' ls ) are respectively converted by momentum encoder to g v (V cls ) and g w (W cls ) is used to encode the image features and text features.
9. The method for identifying critical and severe diseases based on multimodal data according to claim 1, characterized in that: Step S4 also includes setting a predetermined threshold to determine whether the case is an emergency or critical illness, and controlling the sensitivity and specificity of the recognition result by adjusting the threshold; The preprocessed data is input into the trained judgment model to obtain real-time recognition results, and whether it is an emergency or critical case is determined based on the probability or classification result output by the model. The recognition results are fed back to medical staff in real time, and medical staff are reminded to deal with emergency and critical cases in a timely manner through alarm systems, displays or mobile applications.
10. An emergency and critical illness identification system based on multimodal data, characterized in that: include: Data collection module: used to collect multimodal historical medical record data of patients, and to clean and standardize the multimodal historical medical record data; The multimodal historical medical record data includes multiple data types including text, image, sound and physiological signal; Model generation module: used to extract lesion features and key diagnostic information from the multimodal historical medical record data to form a feature set, fuse the feature set using a machine learning model, and generate a judgment model; Model optimization module: used to train the judgment model using the multimodal historical medical record data, optimize model parameters, and evaluate the performance of the judgment model; Result generation module: used to input the patient's real-time medical data into the judgment model to identify potential critical and severe cases and generate early warning signals, notify medical personnel of the identification results, and provide reference knowledge by searching the preset knowledge base to assist doctors in making decisions.
Citation Information
Cited By
Three-branch diagnosis and evaluation method and device based on multi-modal remote medical data
CN119337166A
Three-branch diagnosis and evaluation method and device based on multimodal telemedicine data
CN119337166B
Intelligent medical diagnosis and decision support method using multi-source data
CN119694535A
Dynamic critical disease treatment path construction method and system based on historical medical record analysis
CN119964786A
Radiotherapy scheme decision-making method and system based on tumor classification labeling database
CN121171483A
