Patient data intelligent analysis method and device based on CCE data

Abstract

The invention relates to the technical field of data analysis, and discloses a patient data intelligent analysis method and device based on CCE data, and the method comprises the steps: inputting the multi-modal data of a patient into a feature embedding layer of a trained multi-modal classification early warning model for feature conversion, and obtaining feature embedded data; the multi-modal data comprises text data, physiological data, laboratory examination data and CEC data; inputting the feature embedded data into a feature fusion layer of a multi-modal classification early warning model for cross-modal feature fusion to obtain feature fusion data; inputting the feature fusion data into a feature extraction layer of a multi-modal classification early warning model for feature extraction to obtain a feature extraction result; and inputting the feature extraction result into a classification layer of the multi-modal classification early warning model for classification prediction to obtain a disease early warning result about the target disease. Therefore, the analysis accuracy of the patient data can be improved, so that the early warning accuracy of the disease of the patient is improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a patient data intelligent analysis method and device based on CCE data. Background Art

[0002] In clinical diagnosis, doctors usually analyze the patient's condition by combining various physiological data collected from the patient and the patient's main complaints, in order to provide a treatment plan that suits the patient's actual situation.

[0003] However, in the actual diagnosis process, due to the complex causes of some diseases (such as shock) and the hidden early symptoms, it is difficult to identify the precursor symptoms of these diseases using existing technologies, which can easily lead to missing the early warning of the patient's condition and causing the patient to miss the best time for treatment.

[0004] Therefore, a technical solution is proposed that can improve the analysis accuracy of patient data, thereby improving the accuracy of early warning of patient symptoms. Summary of the invention

[0005] The present invention provides a method and device for intelligent analysis of patient data based on CCE data, which can improve the accuracy of analysis of patient data, thereby improving the accuracy of early warning of patient symptoms.

[0006] In order to solve the above technical problems, the first aspect of the present invention discloses a patient data intelligent analysis method based on CCE data, the method comprising:

[0007] Inputting the multimodal data about the patient into the feature embedding layer of the trained multimodal classification early warning model for feature conversion processing to obtain feature embedding data; the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data;

[0008] Inputting the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data;

[0009] Inputting the feature fusion data into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease to obtain a feature extraction result;

[0010] The feature extraction result is input into the classification layer of the multimodal classification warning model for classification prediction to obtain the symptom warning result of the patient regarding the target symptom.

[0011] As an optional implementation, in the first aspect of the present invention, the multimodal data is determined by:

[0012] Obtaining the medical record information of the patient, and extracting text data corresponding to the patient from the medical record information; the medical record information includes one or more combinations of diagnosis records, comorbidity information, clinical sign information and patient complaint information;

[0013] Based on the monitoring equipment, the physiological data of the patient within a preset time period is collected; the physiological data includes one or more combinations of body temperature data, respiratory data, electrocardiogram data, hemodynamic data and blood oxygen saturation data;

[0014] Based on the collected patient sample material, determining the laboratory test data corresponding to the patient; the laboratory test data includes one or more combinations of blood gas analysis data, blood routine data, blood biochemistry data, D-dimer data, myocardial enzyme data and BNP data;

[0015] Based on the determined cardiac ultrasound digital analysis system, CCE data of the patient is acquired; the CCE data includes CCE image data and / or CCE analysis data.

[0016] As an optional implementation, in the first aspect of the present invention, the feature embedding layer includes a first self-fusion module for the text data and a second self-fusion module for the variable data;

[0017] The step of inputting the multimodal data about the patient into the feature embedding layer of the trained multimodal classification early warning model for feature conversion processing to obtain feature embedding data includes:

[0018] Inputting text data about the patient into the feature embedding layer of the trained multimodal classification warning model, performing a first feature conversion process on the first self-fusion module of the text data, and obtaining text feature embedding data;

[0019] Inputting the variable data about the patient into the feature embedding layer of the multimodal classification early warning model, performing a second feature conversion process on the second self-fusion module of the variable data, and obtaining variable feature embedding data;

[0020] The feature embedding data includes the text feature embedding data and the variable feature embedding data.

[0021] As an optional implementation, in the first aspect of the present invention, the text data about the patient is input into the feature embedding layer of the trained multimodal classification warning model, and the first self-fusion module of the text data performs a first feature conversion process to obtain text feature embedding data, including:

[0022] Inputting text data about the patient into a first self-fusion module for the text data in a feature embedding layer of a trained multimodal classification early warning model;

[0023] Based on the first self-fusion module, the text data is divided into a plurality of text blocks according to the positions of punctuation marks recorded in the text data;

[0024] Based on the first self-fusion module, each of the text blocks is divided into a plurality of character combinations to obtain a character list corresponding to the text data; the character combination includes at least one character;

[0025] Based on the first self-fusion module, generating a token sequence corresponding to the character list according to a preset character recognition method;

[0026] Based on the first self-fusion module, the token sequence is converted into a digital sequence to obtain text feature embedding data corresponding to the text data;

[0027] The step of inputting the variable data about the patient into the feature embedding layer of the multimodal classification early warning model and performing a second feature conversion process on the second self-fusion module of the variable data to obtain variable feature embedding data includes:

[0028] Inputting the variable data about the patient into a second self-fusion module for the variable data in the feature embedding layer of the multimodal classification early warning model;

[0029] Based on the fully connected linear layer set by the second self-fusion module, the variable data is analyzed to obtain variable feature embedding data.

[0030] As an optional implementation, in the first aspect of the present invention, the feature fusion layer includes a plurality of bidirectional multimodal attention modules;

[0031] The step of inputting the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data includes:

[0032] For each of the bidirectional multimodal attention modules in the feature fusion layer of the multimodal classification warning model, the data to be fused is input into the bidirectional multimodal attention module; if the bidirectional multimodal attention module is ranked first, the data to be fused is the feature embedded data; if the bidirectional multimodal attention module is not ranked first, the data to be fused is the fusion result output by the bidirectional multimodal attention module ranked adjacent to the first;

[0033] For each of the bidirectional multimodal attention modules, based on the first linear layer of the bidirectional multimodal attention module, the data to be fused is subjected to layer regularization processing to obtain a first intermediate result; the first intermediate result includes an intermediate text matrix corresponding to the text data to be fused in the data to be fused and an intermediate variable matrix corresponding to the variable data to be fused in the data to be fused;

[0034] For each of the bidirectional multimodal attention modules, based on the bidirectional multimodal attention module, perform an inter-modal information extraction operation on the first intermediate result to obtain a second intermediate result; the second intermediate result includes a second intermediate text result and a second intermediate variable result;

[0035] For each of the bidirectional multimodal attention modules, based on a pre-set residual learning calculation formula, the fusion result corresponding to the bidirectional multimodal attention module is calculated according to the second intermediate result and the data to be fused; if the bidirectional multimodal attention module is ranked last, the fusion result is determined as the feature fusion data.

[0036] As an optional implementation, in the first aspect of the present invention, the feature extraction layer includes a plurality of self-attention modules;

[0037] The step of inputting the feature fusion data into the feature extraction layer of the multimodal classification warning model to extract target disease-oriented features to obtain feature extraction results includes:

[0038] For each of the self-attention modules in the feature extraction layer of the multimodal classification warning model, inputting the data to be extracted into the self-attention module to perform a feature extraction operation for the target disease, and obtaining an attention map corresponding to the self-attention module;

[0039] Among them, if the self-attention module is ranked first, the data to be extracted is the feature fusion data; if the self-attention module is not ranked first, the data to be extracted is the attention map output by the self-attention module ranked first; if the self-attention module is ranked last, the attention map is determined as the feature extraction result.

[0040] As an optional implementation, in the first aspect of the present invention, when the target disease is shock, the disease warning result includes the warning shock type corresponding to the patient and the probability of shock occurrence corresponding to the warning shock type.

[0041] The second aspect of the present invention discloses a patient data intelligent analysis device based on CCE data, the device comprising:

[0042] A feature embedding unit, used for inputting multimodal data about the patient into a feature embedding layer of a trained multimodal classification early warning model for feature conversion processing to obtain feature embedded data; the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data;

[0043] A feature fusion unit, used for inputting the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data;

[0044] A feature extraction unit, used for inputting the feature fusion data into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease and obtain a feature extraction result;

[0045] A classification warning unit is used to input the feature extraction result into the classification layer of the multimodal classification warning model for classification prediction, so as to obtain the symptom warning result of the patient regarding the target symptom.

[0046] As an optional implementation, in the second aspect of the present invention, the multimodal data is determined by:

[0047] Obtaining the medical record information of the patient, and extracting text data corresponding to the patient from the medical record information; the medical record information includes one or more combinations of diagnosis records, comorbidity information, clinical sign information and patient complaint information;

[0048] Based on the monitoring equipment, the physiological data of the patient within a preset time period is collected; the physiological data includes one or more combinations of body temperature data, respiratory data, electrocardiogram data, hemodynamic data and blood oxygen saturation data;

[0049] Based on the collected patient sample material, determining the laboratory test data corresponding to the patient; the laboratory test data includes one or more combinations of blood gas analysis data, blood routine data, blood biochemistry data, D-dimer data, myocardial enzyme data and BNP data;

[0050] Based on the determined cardiac ultrasound digital analysis system, CCE data of the patient is acquired; the CCE data includes CCE image data and / or CCE analysis data.

[0051] As an optional implementation, in the second aspect of the present invention, the feature embedding layer includes a first self-fusion module for the text data and a second self-fusion module for the variable data;

[0052] The feature embedding unit inputs the multimodal data about the patient into the feature embedding layer of the trained multimodal classification warning model for feature conversion processing to obtain the specific method of feature embedding data, including:

[0053] Inputting text data about the patient into the feature embedding layer of the trained multimodal classification warning model, performing a first feature conversion process on the first self-fusion module of the text data, and obtaining text feature embedding data;

[0054] Inputting the variable data about the patient into the feature embedding layer of the multimodal classification early warning model, performing a second feature conversion process on the second self-fusion module of the variable data, and obtaining variable feature embedding data;

[0055] The feature embedding data includes the text feature embedding data and the variable feature embedding data.

[0056] As an optional implementation, in the second aspect of the present invention, the feature embedding unit inputs the text data about the patient into the feature embedding layer of the trained multimodal classification warning model, performs a first feature conversion process on the first self-fusion module of the text data, and obtains a specific method of text feature embedding data, including:

[0057] Inputting text data about the patient into a first self-fusion module for the text data in a feature embedding layer of a trained multimodal classification early warning model;

[0058] Based on the first self-fusion module, the text data is divided into a plurality of text blocks according to the positions of punctuation marks recorded in the text data;

[0059] Based on the first self-fusion module, each of the text blocks is divided into a plurality of character combinations to obtain a character list corresponding to the text data; the character combination includes at least one character;

[0060] Based on the first self-fusion module, generating a token sequence corresponding to the character list according to a preset character recognition method;

[0061] Based on the first self-fusion module, the token sequence is converted into a digital sequence to obtain text feature embedding data corresponding to the text data;

[0062] The feature embedding unit inputs the variable data about the patient into the feature embedding layer of the multimodal classification early warning model, performs a second feature conversion process on the second self-fusion module of the variable data, and obtains the specific method of variable feature embedding data, including:

[0063] Inputting the variable data about the patient into a second self-fusion module for the variable data in the feature embedding layer of the multimodal classification early warning model;

[0064] Based on the fully connected linear layer set by the second self-fusion module, the variable data is analyzed to obtain variable feature embedding data.

[0065] As an optional implementation, in the second aspect of the present invention, the feature fusion layer includes a plurality of bidirectional multimodal attention modules;

[0066] The feature fusion unit inputs the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain the feature fusion data in a specific manner, including:

[0067] For each of the bidirectional multimodal attention modules in the feature fusion layer of the multimodal classification warning model, the data to be fused is input into the bidirectional multimodal attention module; if the bidirectional multimodal attention module is ranked first, the data to be fused is the feature embedded data; if the bidirectional multimodal attention module is not ranked first, the data to be fused is the fusion result output by the bidirectional multimodal attention module ranked adjacent to the first;

[0068] For each of the bidirectional multimodal attention modules, based on the first linear layer of the bidirectional multimodal attention module, the data to be fused is subjected to layer regularization processing to obtain a first intermediate result; the first intermediate result includes an intermediate text matrix corresponding to the text data to be fused in the data to be fused and an intermediate variable matrix corresponding to the variable data to be fused in the data to be fused;

[0069] For each of the bidirectional multimodal attention modules, based on the bidirectional multimodal attention module, perform an inter-modal information extraction operation on the first intermediate result to obtain a second intermediate result; the second intermediate result includes a second intermediate text result and a second intermediate variable result;

[0070] For each of the bidirectional multimodal attention modules, based on a pre-set residual learning calculation formula, the fusion result corresponding to the bidirectional multimodal attention module is calculated according to the second intermediate result and the data to be fused; if the bidirectional multimodal attention module is ranked last, the fusion result is determined as the feature fusion data.

[0071] As an optional implementation, in the second aspect of the present invention, the feature extraction layer includes a plurality of self-attention modules;

[0072] The feature extraction unit inputs the feature fusion data into the feature extraction layer of the multimodal classification warning model to extract the target disease-oriented features, and obtains the feature extraction results in a specific manner, including:

[0073] For each of the self-attention modules in the feature extraction layer of the multimodal classification warning model, inputting the data to be extracted into the self-attention module to perform a feature extraction operation for the target disease, and obtaining an attention map corresponding to the self-attention module;

[0074] Among them, if the self-attention module is ranked first, the data to be extracted is the feature fusion data; if the self-attention module is not ranked first, the data to be extracted is the attention map output by the self-attention module ranked first; if the self-attention module is ranked last, the attention map is determined as the feature extraction result.

[0075] As an optional implementation, in the second aspect of the present invention, when the target disease is shock, the disease warning result includes the warning shock type corresponding to the patient and the probability of shock occurrence corresponding to the warning shock type.

[0076] The third aspect of the present invention discloses another patient data intelligent analysis device based on CCE data, the device comprising:

[0077] A memory storing executable program code;

[0078] a processor coupled to the memory;

[0079] The processor calls the executable program code stored in the memory to execute part or all of the steps of the patient data intelligent analysis method based on CCE data disclosed in the first aspect of the present invention.

[0080] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps of the patient data intelligent analysis method based on CCE data disclosed in the first aspect of the present invention.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] In the present invention, multimodal data about the patient is input into the feature embedding layer of a trained multimodal classification warning model for feature conversion processing to obtain feature embedded data; the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data; the feature embedded data is input into the feature fusion layer of the multimodal classification warning model for cross-modal feature fusion to obtain feature fusion data; the feature fusion data is input into the feature extraction layer of the multimodal classification warning model for feature extraction for the target disease to obtain feature extraction results; the feature extraction results are input into the classification layer of the multimodal classification warning model for classification prediction to obtain the patient's disease warning results for the target disease. It can be seen that the implementation of the present invention can input the multimodal data about the patient into the trained multimodal classification warning model for feature conversion to obtain feature embedding data, and then input the feature embedding data into the feature fusion layer for cross-modal feature fusion to obtain feature fusion data, and then input the feature fusion data into the feature extraction layer for disease-oriented feature extraction to obtain feature extraction results, and finally input the feature extraction results into the classification layer for classification prediction to obtain the disease warning results of the patient about the target disease, and can realize the fusion analysis of the multimodal data containing CCE data, thereby realizing the early warning and classification of the target disease for the patient, and can improve the analysis accuracy and comprehensiveness of the patient data, thereby improving the recognition accuracy of the patient's symptoms, and then improving the analysis accuracy of the patient about the target disease, so as to improve the early warning accuracy and early warning efficiency of the patient's disease, which is conducive to medical staff to take further examination and treatment measures for the patient based on the disease warning results, and is conducive to improving the timeliness of the treatment of the patient's disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 It is a flowchart of a patient data intelligent analysis method based on CCE data disclosed in an embodiment of the present invention;

[0085] Figure 2 It is a flowchart of another patient data intelligent analysis method based on CCE data disclosed in an embodiment of the present invention;

[0086] Figure 3 It is a schematic diagram of the architecture of a multimodal classification warning model disclosed in an embodiment of the present invention;

[0087] Figure 4It is a structural schematic diagram of a patient data intelligent analysis device based on CCE data disclosed in an embodiment of the present invention;

[0088] Figure 5 It is a structural schematic diagram of another patient data intelligent analysis device based on CCE data disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0091] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0092] The present invention discloses a patient data intelligent analysis method and device based on CCE data, which can input multimodal data about a patient into a trained multimodal classification warning model for feature conversion to obtain feature embedding data, then input the feature embedding data into a feature fusion layer for cross-modal feature fusion to obtain feature fusion data, then input the feature fusion data into a feature extraction layer for symptom-oriented feature extraction to obtain feature extraction results, and finally input the feature extraction results into a classification layer for classification prediction to obtain the symptom warning results of the patient about the target symptom, and can realize fusion analysis of multimodal data containing CCE data, thereby realizing early warning and classification of the patient about the target symptom, and can improve the analysis accuracy and comprehensiveness of the patient data, thereby improving the recognition accuracy of the patient's symptoms, and then improving the analysis accuracy of the patient about the target symptom, so as to improve the early warning accuracy and early warning efficiency of the patient's symptom, which is conducive to medical staff taking further examination and treatment measures for the patient according to the symptom warning results, and is conducive to improving the timeliness of the treatment of the patient's symptom. The following are described in detail.

[0093] Embodiment 1

[0094] See also Figure 1 , Figure 1 : is a flow chart of a patient data intelligent analysis method based on CCE data disclosed in an embodiment of the present invention. Figure 1 The described patient data intelligent analysis method based on CCE data can be applied to a patient data intelligent analysis device based on CCE data, which may include one of an intelligent analysis device, an intelligent analysis terminal, an intelligent analysis system and a server, wherein the server may be a local server or a cloud server, which is not limited in the embodiment of the present invention; the method can also be applied to a cardiac ultrasound digital analysis system, which is not limited in the embodiment of the present invention. Figure 1 As shown, the patient data intelligent analysis method based on CCE data may include the following operations:

[0095] 101. Input the multimodal data about the patient into the feature embedding layer of the trained multimodal classification warning model for feature conversion processing to obtain feature embedding data.

[0096] In an embodiment of the present invention, multimodal data may include text data and variable data; wherein the variable data may include physiological data, laboratory test data and CEC data; wherein the multimodal classification warning model may be an artificial intelligence model for clinical application constructed after training and testing based on multiple indicators; wherein the multiple indicators may be a variety of data contained in the above multimodal data, which is not limited in the embodiment of the present invention. wherein the feature embedding layer may be used to convert the multimodal data into a unified feature embedding, so the obtained feature embedding data may be in vector form.

[0097] It should be noted that CCE (Critical Care Echocardiography) is a non-invasive ultrasound examination technology. Since CCE can directly visualize the heart structure and identify various common pathological and physiological changes in critically ill patients, CCE has become an important technical means to assess the condition of critically ill patients.

[0098] In the embodiment of the present invention, the architecture of the exemplary multimodal classification warning model can be found in Figure 3 ,in, Figure 3 Schematic diagram of the architecture of a multi-modal classification warning model disclosed in an embodiment of the present invention. Figure 3 As shown, Figure 3 (a) is the overall architecture of the multimodal classification warning model.

[0099] 102. Input the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data.

[0100] 103. The feature fusion data is input into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease to obtain the feature extraction result.

[0101] In the embodiment of the present invention, the target disease may be any disease whose clinical symptoms are related to cardiac data. For example, the target disease may be shock, which is not limited in the embodiment of the present invention.

[0102] 104. Input the feature extraction result into the classification layer of the multimodal classification warning model for classification prediction to obtain the patient's symptom warning result regarding the target symptom.

[0103] In an embodiment of the present invention, the symptom warning result may include the patient's classification type of the target symptom and the patient's symptom occurrence probability of each classification type, which is not limited in the embodiment of the present invention; wherein the classification layer may include a classification head.

[0104] It can be seen that the method described in the embodiment of the present invention can input the multimodal data about the patient into the trained multimodal classification warning model for feature conversion to obtain feature embedding data, and then input the feature embedding data into the feature fusion layer for cross-modal feature fusion to obtain feature fusion data, and then input the feature fusion data into the feature extraction layer for disease-oriented feature extraction to obtain feature extraction results, and finally input the feature extraction results into the classification layer for classification prediction to obtain the disease warning results of the patient about the target disease, and can realize the fusion analysis of the multimodal data containing CCE data, thereby realizing the early warning and classification of the target disease for the patient, and can improve the analysis accuracy and comprehensiveness of the patient data, thereby improving the recognition accuracy of the patient's symptoms, and then improving the analysis accuracy of the patient about the target disease, so as to improve the early warning accuracy and early warning efficiency of the patient's disease, which is conducive to medical staff to take further examination and treatment measures for the patient based on the disease warning results, and is conducive to improving the timeliness of the treatment of the patient's disease.

[0105] In an optional embodiment, the multimodal data may be determined in the following manner:

[0106] Obtaining the patient's medical record information and extracting the patient's corresponding text data from the medical record information; the medical record information includes one or more combinations of diagnosis records, comorbidity information, clinical signs information and patient's chief complaint information;

[0107] Based on the monitoring equipment, the patient's physiological data within a preset time period is collected; the physiological data includes one or more combinations of body temperature data, respiratory data, electrocardiogram data, hemodynamic data and blood oxygen saturation data;

[0108] Based on the collected patient sample materials, determine the corresponding laboratory test data of the patient; the laboratory test data includes one or more combinations of blood gas analysis data, blood routine data, blood biochemistry data, D-dimer data, myocardial enzyme data and BNP data;

[0109] Based on the determined cardiac ultrasound digital analysis system, CCE data of the patient is acquired; the CCE data includes CCE image data and / or CCE analysis data.

[0110] Exemplarily, the clinical sign information may include cold and wet skin, mottling, decreased urine output, confusion, irritability and other descriptions of clinical signs, which are not limited in the embodiments of the present invention.

[0111] Among them, exemplarily, the monitoring device can be a multi-parameter monitor, or it can be other equipment that can monitor the patient's physiological data, which is not limited in the embodiment of the present invention; optionally, the respiratory data can include respiratory rate and / or respiratory waveform, which is not limited in the embodiment of the present invention; optionally, the electrocardiogram data can include heart rate and / or heart rhythm, which is not limited in the embodiment of the present invention; optionally, the hemodynamic data can include a combination of one or more of systolic pressure, diastolic pressure, mean arterial pressure, and central venous pressure, which is not limited in the embodiment of the present invention.

[0112] Among them, optionally, the blood gas analysis data may include a combination of one or more of serum pH value, PaCO2, PaO2, PaO2 / FiO2, and blood lactate value, and may also include other types of blood gas analysis data, which is not limited in the embodiment of the present invention; the routine blood data may include a combination of one or more of hemoglobin concentration, white blood cell count, and platelet count, and may also include other types of routine blood data, which is not limited in the embodiment of the present invention; the blood biochemistry data may include a combination of one or more of blood creatinine value, urea nitrogen value, alanine aminotransferase value, aspartate aminotransferase value, and lactate dehydrogenase value, and may also include other types of blood biochemistry data, which is not limited in the embodiment of the present invention; the myocardial enzyme data may include a combination of one or more of creatine kinase value, creatine kinase isoenzyme value, and troponin value, and may also include other types of myocardial enzyme data, which is not limited in the embodiment of the present invention; Among them, BNP (Brain Natriuretic Peptide) is a peptide hormone synthesized by the heart.

[0113] Among them, optionally, the CCE analysis data may include LVEF (Left Ventricular Ejection Fractions), LVFAC (Left Ventricular Fractional Area Change), mitral E peak, mitral E' peak, mitral E / E' ratio, aortic VTI (Velocity Time Integral), right ventricular / left ventricular end-diastolic volume ratio, inferior vena cava collapsibility index, TAPSE (Tricuspid Annular Plane Systolic Excursion), RVFAC (Right Ventricular Fractional Area Change) and one or more combinations of right heart morphology assessment, and may also include other data obtained by CCE detection, which is not limited to the embodiments of the present invention.

[0114] Among them, the cardiac ultrasound digital analysis system is a knowledge retrieval and neural graphics network fusion three-dimensional modeling algorithm and electromagnetic tracking system, which can build a real-time, dynamic three-dimensional cardiac cycle full heart model and embed it into the ultrasound section being collected.

[0115] It can be seen that this optional embodiment can extract text data corresponding to the patient from the acquired patient medical record information, collect the patient's physiological data based on the monitoring equipment, and determine the patient's corresponding laboratory test data based on the collected patient sample materials, as well as obtain the patient's CCE data based on the cardiac ultrasound digital analysis system, which can improve the flexibility and reliability of obtaining the patient's multimodal data, and is conducive to improving the accuracy and comprehensiveness of the multimodal data, thereby helping to improve the comprehensiveness and accuracy of the analysis of the patient's data.

[0116] In this optional embodiment, optionally, based on the determined cardiac ultrasound digital analysis system, obtaining the patient's CCE data may include the following operations:

[0117] Obtaining original CCE image data of the patient;

[0118] Determine an optimal denoising threshold corresponding to the original CCE image data based on the determined first preset algorithm corresponding to the cardiac ultrasound digital analysis system;

[0119] Based on a second preset algorithm corresponding to the cardiac ultrasound digital analysis system, and according to an optimal denoising threshold, an image texture denoising operation is performed on the original CCE image data to obtain first denoised image data;

[0120] Based on a third preset algorithm corresponding to the cardiac ultrasound digital analysis system, performing a denoising operation on the first denoised image data to obtain second denoised image data;

[0121] Based on the index evaluation method pre-set by the cardiac ultrasound digital analysis system, an image quality evaluation operation is performed on the second denoised image data to obtain quality optimized image data as the patient's CCE image data;

[0122] Based on the cardiac ultrasound digital analysis system, the CCE image data is analyzed to obtain CCE analysis data.

[0123] It can be seen that this optional embodiment can also determine the optimal denoising threshold corresponding to the original CCE image data of the patient based on the first preset algorithm corresponding to the determined cardiac ultrasound digital analysis system, and then perform image texture denoising on the original CCE image data based on the second preset algorithm and according to the optimal denoising threshold to obtain first denoised image data, and then generate denoising on the first denoised image data based on the third preset algorithm to obtain second denoised image data, and then perform image quality evaluation on the second denoised image data based on the index evaluation method to obtain quality optimized image data as the patient's CCE image data, thereby improving the denoising accuracy of the CCE image, which is beneficial to effectively retain the detailed features of the CCE image, thereby facilitating improving the quality of the CCE image, and further facilitating improving the analysis accuracy of the CCE image; and, it is also possible to obtain CCE analysis data by analyzing the CCE image data, and it is possible to obtain CCE analysis data by analyzing the CCE image data with optimized image quality, which is beneficial to improving the accuracy of the CCE analysis data.

[0124] In this optional embodiment, optionally, the feature embedding layer includes a first self-fusion module for text data and a second self-fusion module for variable data;

[0125] The multimodal data about the patient is input into the feature embedding layer of the trained multimodal classification warning model for feature conversion processing to obtain feature embedding data, which may include the following operations:

[0126] Inputting the text data about the patient into the feature embedding layer of the trained multimodal classification warning model, performing a first feature conversion process on the first self-fusion module of the text data, and obtaining text feature embedding data;

[0127] Inputting the variable data about the patient into the feature embedding layer of the trained multimodal classification early warning model, performing a second feature conversion process on the second self-fusion module of the variable data, and obtaining variable feature embedding data;

[0128] Among them, feature embedding data includes text feature embedding data and variable feature embedding data.

[0129] Wherein, optionally, the first self-fusion module for text data can be a Tokenizer self-fusion module for text data, and the second self-fusion module for variable data can be a fully connected layer linear Tokenizer self-fusion module for variable data, which is not limited in the embodiment of the present invention. Wherein, the Tokenizer self-fusion module can be used to convert raw data into a digital sequence that can be understood by the model, which is not limited in the embodiment of the present invention. Further, exemplarily, the Tokenizer self-fusion module for text data can be a BERT Tokenizer pre-trained based on a large corpus, which is not limited in the embodiment of the present invention.

[0130] It can be seen that this optional embodiment can also input the text data about the patient into the first self-fusion module for text data in the multimodal classification warning model to perform a first feature conversion process on the text data to obtain text feature embedding data, and input the variable data about the patient into the second self-fusion module for variable data to perform a second feature conversion process on the variable data to obtain variable feature embedding data. It can use self-fusion modules with different structures to perform different feature conversion processes on different types of patient data, which can improve the feature conversion flexibility and feature conversion accuracy of multimodal data, thereby helping to improve the accuracy of feature embedded data.

[0131] In this optional embodiment, optionally, inputting the text data about the patient into the feature embedding layer of the trained multimodal classification warning model to perform a first feature conversion process on the first self-fusion module of the text data to obtain text feature embedding data may include the following operations:

[0132] Inputting text data about the patient into the first self-fusion module for text data in the feature embedding layer of the trained multimodal classification early warning model;

[0133] Based on the first self-fusion module, the text data is divided into a plurality of text blocks according to the positions of punctuation marks recorded in the text data;

[0134] Based on the first self-fusion module, each text block is divided into a plurality of character combinations to obtain a character list corresponding to the text data; the character combination includes at least one character;

[0135] Based on the first self-fusion module, a token sequence corresponding to the character list is generated according to a preset character recognition method;

[0136] Based on the first self-fusion module, the token sequence is converted into a digital sequence to obtain text feature embedding data corresponding to the text data.

[0137] Among them, each text block includes multiple characters; optionally, the character type of the characters included in each text block can be any text type. For example, the text type can be Chinese type or other text types, which is not limited in the embodiment of the present invention.

[0138] Wherein, the token sequence includes multiple tokens; optionally, each token may include at least one character and / or word and / or word and / or phrase and / or statement and / or sentence, which is not limited in the embodiment of the present invention; wherein each token may correspond to a character combination with actual semantics. Optionally, based on the first self-fusion module, according to a pre-set character recognition method, a token sequence corresponding to the character list is generated, which can be specifically as follows: based on the first self-fusion module, each character in the character list is first recognized to obtain a character recognition result; then, according to the character recognition result and based on a pre-set word segmentation rule and / or dictionary, multiple tokens are extracted from the character list in sequence to obtain a token sequence, which is not limited in the embodiment of the present invention.

[0139] It can be seen that this optional embodiment can also input text data about the patient into the first self-fusion module for text data in the feature embedding layer of the multimodal classification warning model, and based on the first self-fusion module, divide the text data into multiple text blocks according to the punctuation positions recorded in the text data, and then divide each text block into multiple character combinations to obtain a character list, and then generate a token sequence corresponding to the character list according to a pre-set character recognition method, and then convert the token sequence into a digital sequence to obtain text feature embedding data, which can improve the processing accuracy and efficiency of text data, thereby improving the accuracy of converting text data into text feature embedding data.

[0140] In this optional embodiment, optionally, inputting the variable data about the patient into the feature embedding layer of the trained multimodal classification early warning model to perform a second feature conversion process on the second self-fusion module of the variable data to obtain the variable feature embedding data may include the following operations:

[0141] Inputting the variable data about the patient into the second self-fusion module for the variable data in the feature embedding layer of the trained multimodal classification early warning model;

[0142] Based on the fully connected linear layer set by the second self-fusion module, the variable data is analyzed to obtain variable feature embedding data.

[0143] Among them, the fully connected linear layer set by the second self-fusion module can be a learnable fully connected linear layer, which is not limited in the embodiment of the present invention.

[0144] Among them, optionally, the calculation formula for analyzing variable data in the fully connected linear layer set by the second self-fusion module can be as follows:

[0145] y=Wx+b

[0146] Among them, x is the input feature vector, which can be the input vector corresponding to each data contained in the variable data; W is the weight matrix, b is the bias vector, and y is the output feature vector, which can be the output vector corresponding to each data contained in the variable data; wherein, optionally, W and b are learnable parameters that can be learned in the overall optimization of the model.

[0147] Among them, optionally, the variable feature embedded data includes an output vector corresponding to each data contained in the variable data, that is, the variable feature embedded data can be in vector form, which is not limited in the embodiment of the present invention.

[0148] It can be seen that this optional embodiment can also input the variable data about the patient into the second self-fusion module for the variable data in the feature embedding layer of the multimodal classification early warning model, and analyze the variable data based on the fully connected linear layer set by the second self-fusion module to obtain variable feature embedding data, which can improve the processing accuracy and efficiency of the variable data, thereby improving the accuracy of converting the variable data into variable feature embedding data.

[0149] Embodiment 2

[0150] See also Figure 2 , Figure 2 : is a flow chart of a patient data intelligent analysis method based on CCE data disclosed in an embodiment of the present invention. Figure 2 The described patient data intelligent analysis method based on CCE data can be applied to a patient data intelligent analysis device based on CCE data, which may include one of an intelligent analysis device, an intelligent analysis terminal, an intelligent analysis system and a server, wherein the server may be a local server or a cloud server, which is not limited in the embodiment of the present invention; the method can also be applied to a cardiac ultrasound digital analysis system, which is not limited in the embodiment of the present invention. Figure 2 As shown, the patient data intelligent analysis method based on CCE data may include the following operations:

[0151] 201. Input the multimodal data about the patient into the feature embedding layer of the trained multimodal classification warning model for feature conversion processing to obtain feature embedding data.

[0152] In the embodiment of the present invention, the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data.

[0153] In an embodiment of the present invention, the feature fusion layer of the multimodal classification warning model includes multiple bidirectional multimodal attention modules; exemplarily, the number of bidirectional multimodal attention modules in the feature fusion layer can be two or other numbers, which is not limited by the embodiment of the present invention.

[0154] 202. For each bidirectional multimodal attention module in the feature fusion layer of the multimodal classification warning model, input the data to be fused into the bidirectional multimodal attention module.

[0155] In an embodiment of the present invention, if the bidirectional multimodal attention module is ranked first, the data to be fused is the feature embedded data; if the bidirectional multimodal attention module is not ranked first, the data to be fused is the fusion result output by the bidirectional multimodal attention module ranked adjacent to the first one.

[0156] 203. For each bidirectional multimodal attention module, based on the first linear layer of the bidirectional multimodal attention module, perform layer regularization processing on the data to be fused to obtain a first intermediate result.

[0157] In an embodiment of the present invention, the first intermediate result includes an intermediate text matrix corresponding to the text data to be fused in the data to be fused and an intermediate variable matrix corresponding to the variable data to be fused in the data to be fused; wherein the intermediate text matrix may include a text query matrix, a text key matrix and a text value matrix; the intermediate variable matrix may include a variable query matrix, a variable key matrix and a variable value matrix; optionally, based on the regularization calculation formula corresponding to the above-mentioned first linear layer, layer regularization processing may be performed on the data to be fused to obtain the first intermediate result, which is not limited in the embodiment of the present invention.

[0158] 204. For each bidirectional multimodal attention module, based on the bidirectional multimodal attention module, perform an inter-modal information extraction operation on the first intermediate result to obtain a second intermediate result.

[0159] In an embodiment of the present invention, the second intermediate result includes a second intermediate text result and a second intermediate variable result; optionally, based on the cross-modal calculation formula of the bidirectional multimodal attention module, an inter-modal information extraction operation can be performed on the first intermediate result to obtain a second intermediate result, which is not limited in the embodiment of the present invention.

[0160] 205. For each bidirectional multimodal attention module, based on a preset residual learning calculation formula, according to the second intermediate result and the data to be fused, calculate the fusion result corresponding to the bidirectional multimodal attention module.

[0161] In an embodiment of the present invention, if the bidirectional multimodal attention module is ranked last, the fusion result is determined as feature fusion data; wherein the fusion result includes a text fusion result and a variable fusion result.

[0162] In the embodiment of the present invention, exemplarily, in the bidirectional multimodal attention module, for the text feature embedding data X in the input data to be fused T and variable feature embedding data X V , first use the first linear layer and layer normalization processing (Layer Normalization), project the two embedded data according to the following regularization calculation formula, and obtain the corresponding query matrix (Query), key matrix (Key) and value matrix (Value):

[0163]

[0164] Among them, l is the sorting number of the bidirectional multimodal attention module, is the text query matrix, is the text key matrix, is a matrix of text values, is the variable query matrix, is the variable key matrix, is the variable value matrix.

[0165] Next, the cross-modal calculation formula of bidirectional multimodal attention processing is used to extract inter-modal information and obtain the second intermediate result:

[0166]

[0167] in:

[0168]

[0169] Among them, λ is the weighted hyperparameter; is the second intermediate text result, is the second intermediate variable result.

[0170] Then, using the idea of ​​residual learning and based on the residual learning calculation formula, the second intermediate result is input into the layer regularization processing and MLP (Multilayer Perceptron) of the second linear layer, and the processing result is added to the original input to finally obtain the output of this layer and the input of the next layer, that is, the fusion result:

[0171]

[0172] in, is the text fusion result, is the variable fusion result.

[0173] Exemplarily, the structure of the bidirectional multimodal attention module included in the feature fusion layer of the multimodal classification warning model can be as follows: Figure 3(b) is shown, and the embodiment of the present invention is not limited thereto.

[0174] 206. The feature fusion data is input into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease to obtain a feature extraction result.

[0175] 207. Input the feature extraction result into the classification layer of the multimodal classification warning model for classification prediction to obtain the patient's symptom warning result regarding the target symptom.

[0176] In the embodiment of the present invention, for other detailed descriptions of step 201 and step 206-step 207, please refer to the detailed descriptions of step 101 and step 103-step 104 in the first embodiment, and the embodiment of the present invention will not be repeated here.

[0177] It can be seen that the method described in the embodiment of the present invention can input the multimodal data about the patient into the trained multimodal classification warning model for feature conversion to obtain feature embedding data, and then input the feature embedding data into the feature fusion layer for cross-modal feature fusion to obtain feature fusion data, and then input the feature fusion data into the feature extraction layer for disease-oriented feature extraction to obtain feature extraction results, and finally input the feature extraction results into the classification layer for classification prediction to obtain the disease warning results of the patient about the target disease, and can realize the fusion analysis of the multimodal data containing CCE data, thereby realizing the early warning and classification of the target disease for the patient, and can improve the analysis accuracy and comprehensiveness of the patient data, thereby improving the recognition accuracy of the patient's symptoms, and then improving the analysis accuracy of the patient about the target disease, so as to improve the early warning accuracy and early warning efficiency of the patient's disease, which is conducive to medical staff to take further examination and treatment measures for the patient based on the disease warning results, and is conducive to improving the timeliness of the treatment of the patient's disease. In addition, for each bidirectional multimodal attention module in the feature fusion layer of the multimodal classification warning model, the data to be fused can be input into the bidirectional multimodal attention module, and based on the first linear layer, the data to be fused is subjected to layer regularization processing to obtain a first intermediate result, and then the inter-modal information extraction operation is performed on the first intermediate result to obtain a second intermediate result, and then based on the residual learning calculation formula and according to the second intermediate result and the data to be fused, the fusion result corresponding to the bidirectional multimodal attention module is calculated, and the mutual correlation between different modalities can be discovered and encoded through the bidirectional multimodal attention mechanism, so as to realize cross-modal feature fusion between multimodal data, and improve the accuracy of feature analysis and feature fusion between different modal data, which is beneficial to improve the accuracy of feature fusion data, and then is beneficial to improve the accuracy of subsequent feature extraction.

[0178] In an optional embodiment, the feature extraction layer includes a plurality of self-attention modules;

[0179] Among them, the feature fusion data is input into the feature extraction layer of the multimodal classification warning model to extract the features of the target disease to obtain the feature extraction results, which may include the following operations:

[0180] For each self-attention module in the feature extraction layer of the multimodal classification warning model, the data to be extracted is input into the self-attention module to perform a feature extraction operation for the target disease, and an attention map corresponding to the self-attention module is obtained;

[0181] Among them, if the self-attention module is ranked first, the data to be extracted is the feature fusion data; if the self-attention module is not ranked first, the data to be extracted is the attention map output by the self-attention module ranked first; if the self-attention module is ranked last, the attention map is determined as the feature extraction result.

[0182] Exemplarily, the structure of the self-attention module included in the feature extraction layer of the multimodal classification warning model can be as follows: Figure 3 (c) is shown, and the embodiment of the present invention is not limited thereto.

[0183] It can be seen that this optional embodiment can target each self-attention module in the feature extraction layer of the multimodal classification warning model, input the data to be extracted into the self-attention module to perform feature extraction operations for the target disease, obtain the attention map corresponding to the self-attention module, and determine the attention map output by the last-ranked self-attention module as the feature extraction result, which can realize the connection between the modality-independent feature expression and the overall diagnosis-oriented representation, improve the accuracy of the analysis of the patient's symptoms, and thus help improve the accuracy of the analysis of the patient's target disease.

[0184] In another optional embodiment, when the target disease is shock, the disease warning result includes the warning shock type corresponding to the patient and the probability of shock occurrence corresponding to the warning shock type.

[0185] Among them, each warning shock type can be one of a hypovolemic shock type, a cardiogenic shock type, an obstructive shock type, and a distributive shock type, which is not limited in the embodiment of the present invention.

[0186] It can be seen that, when the target disease is shock, the disease warning result can include the patient's corresponding warning shock type and the corresponding probability of shock occurrence, which can improve the accuracy of the disease warning result, thereby facilitating further improving the warning accuracy of the patient's disease, and facilitating medical staff to take further examination and treatment measures for the patient based on the disease warning results.

[0187] In the embodiment of the present invention, the architecture of the exemplary multimodal classification warning model can be found in Figure 3 ,in, Figure 3 is a schematic diagram of the architecture of a multimodal classification warning model disclosed in an embodiment of the present invention. Figure 3 As shown:

[0188] Figure 3 (a) is the overall architecture of the multimodal classification warning model, where: Figure 3 In (a), Text embedding is the first self-fusion module of text embedding, Variable embedding is the second self-fusion module of variable embedding, Bidirectional multimodal attention block is the bidirectional multimodal attention module, Self-attention block is the self-attention module, Classification head is the classification head in the classification layer, and Predict is the disease warning result;

[0189] Figure 3 (b) is the bidirectional multimodal attention module structure included in the feature fusion layer of the multimodal classification warning model, where: Figure 3 (b) Bidirectional multimodal attention is bidirectional multimodal attention processing, Norm is regularization processing, and MLP is multi-layer perceptron;

[0190] Figure 3 (c) is the structure of the self-attention module contained in the feature extraction layer of the multimodal classification warning model, where Self-attention is the self-attention processing and Norm is the regularization processing;

[0191] in, Figure 3 The arrow in the figure indicates the direction of data transmission.

[0192] Embodiment 3

[0193] See also Figure 4 , Figure 4 1 is a schematic diagram of a patient data intelligent analysis device based on CCE data disclosed in an embodiment of the present invention. Figure 4 The described patient data intelligent analysis device based on CCE data may include one of an intelligent analysis device, an intelligent analysis terminal, an intelligent analysis system and a server, wherein the server may be a local server or a cloud server, which is not limited in the embodiments of the present invention; the device can be applied to a cardiac ultrasound digital analysis system, which is not limited in the embodiments of the present invention.

[0194] like Figure 4 As shown, the patient data intelligent analysis device based on CCE data may include:

[0195] The feature embedding unit 301 is used to input the multimodal data about the patient into the feature embedding layer of the trained multimodal classification warning model for feature conversion processing to obtain feature embedded data; the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data;

[0196] A feature fusion unit 302 is used to input the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data;

[0197] The feature extraction unit 303 is used to input the feature fusion data into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease and obtain a feature extraction result;

[0198] The classification warning unit 304 is used to input the feature extraction result into the classification layer of the multimodal classification warning model for classification prediction, so as to obtain the symptom warning result of the patient regarding the target symptom.

[0199] It can be seen that the device described in the embodiment of the present invention can input the multimodal data about the patient into the trained multimodal classification warning model for feature conversion to obtain feature embedding data, and then input the feature embedding data into the feature fusion layer for cross-modal feature fusion to obtain feature fusion data, and then input the feature fusion data into the feature extraction layer for disease-oriented feature extraction to obtain feature extraction results, and finally input the feature extraction results into the classification layer for classification prediction to obtain the disease warning results of the patient about the target disease, and can realize the fusion analysis of the multimodal data containing CCE data, so as to realize the warning and classification of the target disease for the patient, and can improve the accuracy and comprehensiveness of the analysis of the patient data, thereby improving the accuracy of the recognition of the patient's symptoms, and then improving the accuracy of the analysis of the patient's target disease, so as to improve the accuracy and efficiency of the warning of the patient's disease, which is conducive to medical staff to take further examination and treatment measures for the patient based on the disease warning results, and is conducive to improving the timeliness of the treatment of the patient's disease.

[0200] In an optional embodiment, the multimodal data may be determined in the following manner:

[0201] Obtaining the patient's medical record information and extracting the patient's corresponding text data from the medical record information; the medical record information includes one or more combinations of diagnosis records, comorbidity information, clinical signs information and patient's chief complaint information;

[0202] Based on the monitoring equipment, the patient's physiological data within a preset time period is collected; the physiological data includes one or more combinations of body temperature data, respiratory data, electrocardiogram data, hemodynamic data and blood oxygen saturation data;

[0203] Based on the collected patient sample materials, determine the corresponding laboratory test data of the patient; the laboratory test data includes one or more combinations of blood gas analysis data, blood routine data, blood biochemistry data, D-dimer data, myocardial enzyme data and BNP data;

[0204] Based on the determined cardiac ultrasound digital analysis system, CCE data of the patient is acquired; the CCE data includes CCE image data and / or CCE analysis data.

[0205] It can be seen that the device described in the implementation of this optional embodiment can extract text data corresponding to the patient from the acquired patient's medical record information, collect the patient's physiological data based on the monitoring equipment, and determine the patient's corresponding laboratory test data based on the collected patient sample materials, as well as obtain the patient's CCE data based on the cardiac ultrasound digital analysis system, which can improve the flexibility and reliability of obtaining the patient's multimodal data, and is conducive to improving the accuracy and comprehensiveness of the multimodal data, thereby helping to improve the comprehensiveness and accuracy of the analysis of the patient's data.

[0206] In this optional embodiment, optionally, the feature embedding layer includes a first self-fusion module for text data and a second self-fusion module for variable data;

[0207] The feature embedding unit 301 inputs the multimodal data about the patient into the feature embedding layer of the trained multimodal classification warning model for feature conversion processing, and the specific method of obtaining the feature embedded data may include:

[0208] Inputting the text data about the patient into the feature embedding layer of the trained multimodal classification warning model, performing a first feature conversion process on the first self-fusion module of the text data, and obtaining text feature embedding data;

[0209] Inputting the variable data about the patient into the feature embedding layer of the trained multimodal classification early warning model, performing a second feature conversion process on the second self-fusion module of the variable data, and obtaining variable feature embedding data;

[0210] Among them, feature embedding data includes text feature embedding data and variable feature embedding data.

[0211] It can be seen that the device described in the implementation of this optional embodiment can also input the text data about the patient into the first self-fusion module for text data in the multimodal classification warning model to perform a first feature conversion process on the text data to obtain text feature embedding data, and input the variable data about the patient into the second self-fusion module for variable data to perform a second feature conversion process on the variable data to obtain variable feature embedding data. It can use self-fusion modules with different structures to perform different feature conversion processes on different types of patient data, which can improve the feature conversion flexibility and feature conversion accuracy of multimodal data, thereby helping to improve the accuracy of feature embedded data.

[0212] In this optional embodiment, optionally, the feature embedding unit 301 inputs the text data about the patient into the feature embedding layer of the trained multimodal classification warning model to perform a first feature conversion process on the first self-fusion module of the text data to obtain the specific method of text feature embedding data, which may include:

[0213] Inputting text data about the patient into the first self-fusion module for text data in the feature embedding layer of the trained multimodal classification early warning model;

[0214] Based on the first self-fusion module, the text data is divided into a plurality of text blocks according to the positions of punctuation marks recorded in the text data;

[0215] Based on the first self-fusion module, each text block is divided into a plurality of character combinations to obtain a character list corresponding to the text data; the character combination includes at least one character;

[0216] Based on the first self-fusion module, a token sequence corresponding to the character list is generated according to a preset character recognition method;

[0217] Based on the first self-fusion module, the token sequence is converted into a digital sequence to obtain text feature embedding data corresponding to the text data;

[0218] The feature embedding unit 301 inputs the variable data about the patient into the feature embedding layer of the trained multimodal classification warning model, performs a second feature conversion process on the second self-fusion module of the variable data, and obtains the specific method of variable feature embedding data, which may include:

[0219] Inputting the variable data about the patient into the second self-fusion module for the variable data in the feature embedding layer of the trained multimodal classification early warning model;

[0220] Based on the fully connected linear layer set by the second self-fusion module, the variable data is analyzed to obtain variable feature embedding data.

[0221] It can be seen that the device described in the implementation of this optional embodiment can also input the text data about the patient into the first self-fusion module for text data in the feature embedding layer of the multimodal classification warning model, and based on the first self-fusion module, according to the punctuation mark position recorded in the text data, divide the text data into multiple text blocks, and then divide each text block into multiple character combinations to obtain a character list, and then generate a token sequence corresponding to the character list according to a preset character recognition method, and then convert the token sequence into a digital sequence to obtain text feature embedding data, which can improve the processing accuracy and processing efficiency of the text data, thereby improving the accuracy of converting the text data into text feature embedding data; and, it can also input the variable data about the patient into the second self-fusion module for variable data in the feature embedding layer of the multimodal classification warning model, and analyze the variable data based on the fully connected linear layer set by the second self-fusion module to obtain variable feature embedding data, which can improve the processing accuracy and processing efficiency of the variable data, thereby improving the accuracy of converting the variable data into variable feature embedding data. .

[0222] In another optional embodiment, the feature fusion layer includes a plurality of bidirectional multimodal attention modules;

[0223] The feature fusion unit 302 inputs the feature embedding data into the feature fusion layer of the multimodal classification warning model for cross-modal feature fusion to obtain the feature fusion data in a specific manner, which may include:

[0224] For each bidirectional multimodal attention module in the feature fusion layer of the multimodal classification warning model, the data to be fused is input into the bidirectional multimodal attention module; if the bidirectional multimodal attention module is ranked first, the data to be fused is the feature embedded data; if the bidirectional multimodal attention module is not ranked first, the data to be fused is the fusion result output by the bidirectional multimodal attention module ranked adjacently first;

[0225] For each bidirectional multimodal attention module, based on the first linear layer of the bidirectional multimodal attention module, layer regularization processing is performed on the data to be fused to obtain a first intermediate result; the first intermediate result includes an intermediate text matrix corresponding to the text data to be fused in the data to be fused and an intermediate variable matrix corresponding to the variable data to be fused in the data to be fused;

[0226] For each bidirectional multimodal attention module, based on the bidirectional multimodal attention module, perform an inter-modal information extraction operation on the first intermediate result to obtain a second intermediate result; the second intermediate result includes a second intermediate text result and a second intermediate variable result;

[0227] For each bidirectional multimodal attention module, based on the pre-set residual learning calculation formula, the fusion result corresponding to the bidirectional multimodal attention module is calculated according to the second intermediate result and the data to be fused; if the bidirectional multimodal attention module is ranked last, the fusion result is determined as the feature fusion data.

[0228] It can be seen that the device described in the implementation of this optional embodiment can input the data to be fused into the bidirectional multimodal attention module for each bidirectional multimodal attention module in the feature fusion layer of the multimodal classification warning model, and based on the first linear layer, perform layer regularization processing on the data to be fused to obtain a first intermediate result, and then perform an inter-modal information extraction operation on the first intermediate result to obtain a second intermediate result, and then calculate the fusion result corresponding to the bidirectional multimodal attention module based on the residual learning calculation formula and according to the second intermediate result and the data to be fused. The bidirectional multimodal attention mechanism can discover and encode the mutual correlation between different modalities, realize cross-modal feature fusion between multimodal data, improve the accuracy of feature analysis and feature fusion between different modal data, which is beneficial to improving the accuracy of feature fusion data, and then is beneficial to improving the accuracy of subsequent feature extraction.

[0229] In this optional embodiment, optionally, the feature extraction layer includes a plurality of self-attention modules;

[0230] The feature extraction unit 303 inputs the feature fusion data into the feature extraction layer of the multimodal classification warning model to extract the target disease-oriented features. The specific method of obtaining the feature extraction result may include:

[0231] For each self-attention module in the feature extraction layer of the multimodal classification warning model, the data to be extracted is input into the self-attention module to perform a feature extraction operation for the target disease, and an attention map corresponding to the self-attention module is obtained;

[0232] Among them, if the self-attention module is ranked first, the data to be extracted is the feature fusion data; if the self-attention module is not ranked first, the data to be extracted is the attention map output by the self-attention module ranked first; if the self-attention module is ranked last, the attention map is determined as the feature extraction result.

[0233] It can be seen that the device described in the implementation of this optional embodiment can also input the data to be extracted into each self-attention module in the feature extraction layer of the multimodal classification warning model to perform a feature extraction operation for the target disease, obtain the attention map corresponding to the self-attention module, and determine the attention map output by the last-ranked self-attention module as the feature extraction result, which can realize the connection between the modality-independent feature expression and the overall diagnosis-oriented representation, improve the accuracy of the analysis of the patient's symptoms, and thus help improve the accuracy of the analysis of the patient's target disease.

[0234] In yet another optional embodiment, when the target disease is shock, the disease warning result includes the warning shock type corresponding to the patient and the probability of shock occurrence corresponding to the warning shock type.

[0235] It can be seen that when the target disease is shock, the device described in the optional embodiment can provide a disease warning result that includes the patient's corresponding warning shock type and the corresponding probability of shock occurrence, thereby improving the accuracy of the disease warning result, thereby facilitating further improvement of the warning accuracy of the patient's disease, and facilitating medical staff to take further examination and treatment measures on the patient based on the disease warning result.

[0236] Embodiment 4

[0237] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of the structure of another patient data intelligent analysis device based on CCE data disclosed in an embodiment of the present invention. Figure 5 As shown, the patient data intelligent analysis device based on CCE data may include:

[0238] A memory 401 storing executable program codes;

[0239] a processor 402 coupled to the memory 401;

[0240] The processor 402 calls the executable program code stored in the memory 401 to execute part or all of the steps in the patient data intelligent analysis method based on CCE data described in the first embodiment of the present invention or the second embodiment of the present invention.

[0241] Embodiment 5

[0242] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the patient data intelligent analysis method based on CCE data described in Embodiment 1 or Embodiment 2 of the present invention.

[0243] Embodiment 6

[0244] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the patient data intelligent analysis method based on CCE data described in Example 1 or Example 2.

[0245] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0246] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0247] Finally, it should be noted that the patient data intelligent analysis method and device based on CCE data disclosed in the embodiment of the present invention discloses only the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that it is still possible to modify the technical solutions recorded in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A patient data intelligent analysis method based on CCE data, characterized in that: The method comprises: Inputting the multimodal data about the patient into the feature embedding layer of the trained multimodal classification early warning model for feature conversion processing to obtain feature embedding data; the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data; Inputting the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data; Inputting the feature fusion data into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease to obtain a feature extraction result; The feature extraction result is input into the classification layer of the multimodal classification warning model for classification prediction to obtain the symptom warning result of the patient regarding the target symptom.

2. The patient data intelligent analysis method based on CCE data according to claim 1, characterized in that: The multimodal data is determined by: Obtaining the medical record information of the patient, and extracting text data corresponding to the patient from the medical record information; the medical record information includes one or more combinations of diagnosis records, comorbidity information, clinical sign information and patient complaint information; Based on the monitoring equipment, collecting the physiological data of the patient within a preset time period; The physiological data includes one or more combinations of body temperature data, respiratory data, electrocardiogram data, hemodynamic data and blood oxygen saturation data; Based on the collected patient sample material, determining the laboratory test data corresponding to the patient; the laboratory test data includes one or more combinations of blood gas analysis data, blood routine data, blood biochemistry data, D-dimer data, myocardial enzyme data and BNP data; Based on the determined cardiac ultrasound digital analysis system, CCE data of the patient is acquired; the CCE data includes CCE image data and / or CCE analysis data.

3. The patient data intelligent analysis method based on CCE data according to claim 2 is characterized in that: The feature embedding layer includes a first self-fusion module for the text data and a second self-fusion module for the variable data; The step of inputting the multimodal data about the patient into the feature embedding layer of the trained multimodal classification early warning model for feature conversion processing to obtain feature embedding data includes: Inputting text data about the patient into the feature embedding layer of the trained multimodal classification warning model, performing a first feature conversion process on the first self-fusion module of the text data, and obtaining text feature embedding data; Inputting the variable data about the patient into the feature embedding layer of the multimodal classification early warning model, performing a second feature conversion process on the second self-fusion module of the variable data, and obtaining variable feature embedding data; The feature embedding data includes the text feature embedding data and the variable feature embedding data.

4. The patient data intelligent analysis method based on CCE data according to claim 3 is characterized in that: The step of inputting the text data about the patient into the feature embedding layer of the trained multimodal classification warning model and performing a first feature conversion process on the first self-fusion module of the text data to obtain text feature embedding data includes: Inputting text data about the patient into a first self-fusion module for the text data in a feature embedding layer of a trained multimodal classification early warning model; Based on the first self-fusion module, the text data is divided into a plurality of text blocks according to the positions of punctuation marks recorded in the text data; Based on the first self-fusion module, each of the text blocks is divided into a plurality of character combinations to obtain a character list corresponding to the text data; the character combination includes at least one character; Based on the first self-fusion module, generating a token sequence corresponding to the character list according to a preset character recognition method; Based on the first self-fusion module, the token sequence is converted into a digital sequence to obtain text feature embedding data corresponding to the text data; The step of inputting the variable data about the patient into the feature embedding layer of the multimodal classification early warning model and performing a second feature conversion process on the second self-fusion module of the variable data to obtain variable feature embedding data includes: Inputting the variable data about the patient into a second self-fusion module for the variable data in the feature embedding layer of the multimodal classification early warning model; Based on the fully connected linear layer set by the second self-fusion module, the variable data is analyzed to obtain variable feature embedding data.

5. The patient data intelligent analysis method based on CCE data according to any one of claims 1 to 4, characterized in that: The feature fusion layer includes a plurality of bidirectional multimodal attention modules; The step of inputting the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data includes: For each of the bidirectional multimodal attention modules in the feature fusion layer of the multimodal classification warning model, the data to be fused is input into the bidirectional multimodal attention module; if the bidirectional multimodal attention module is ranked first, the data to be fused is the feature embedded data; if the bidirectional multimodal attention module is not ranked first, the data to be fused is the fusion result output by the bidirectional multimodal attention module ranked adjacent to the first; For each of the bidirectional multimodal attention modules, based on the first linear layer of the bidirectional multimodal attention module, the data to be fused is subjected to layer regularization processing to obtain a first intermediate result; the first intermediate result includes an intermediate text matrix corresponding to the text data to be fused in the data to be fused and an intermediate variable matrix corresponding to the variable data to be fused in the data to be fused; For each of the bidirectional multimodal attention modules, based on the bidirectional multimodal attention module, perform an inter-modal information extraction operation on the first intermediate result to obtain a second intermediate result; the second intermediate result includes a second intermediate text result and a second intermediate variable result; For each of the bidirectional multimodal attention modules, based on a pre-set residual learning calculation formula, the fusion result corresponding to the bidirectional multimodal attention module is calculated according to the second intermediate result and the data to be fused; if the bidirectional multimodal attention module is ranked last, the fusion result is determined as the feature fusion data.

6. The patient data intelligent analysis method based on CCE data according to claim 5, characterized in that: The feature extraction layer includes a plurality of self-attention modules; The step of inputting the feature fusion data into the feature extraction layer of the multimodal classification warning model to extract target disease-oriented features to obtain feature extraction results includes: For each of the self-attention modules in the feature extraction layer of the multimodal classification warning model, inputting the data to be extracted into the self-attention module to perform a feature extraction operation for the target disease, and obtaining an attention map corresponding to the self-attention module; Among them, if the self-attention module is ranked first, the data to be extracted is the feature fusion data; if the self-attention module is not ranked first, the data to be extracted is the attention map output by the self-attention module ranked first; if the self-attention module is ranked last, the attention map is determined as the feature extraction result.

7. The method for intelligent analysis of patient data based on CCE data according to any one of claims 1, 2, 3, 4 and 6, characterized in that: When the target disease is shock, the disease warning result includes the warning shock type corresponding to the patient and the probability of shock occurrence corresponding to the warning shock type.

8. A patient data intelligent analysis device based on CCE data, characterized in that: The device comprises: A feature embedding unit, used for inputting multimodal data about the patient into a feature embedding layer of a trained multimodal classification early warning model for feature conversion processing to obtain feature embedded data; the multimodal data includes text data and variable data; the variable data includes physiological data, laboratory test data and CEC data; A feature fusion unit, used for inputting the feature embedding data into the feature fusion layer of the multimodal classification warning model to perform cross-modal feature fusion to obtain feature fusion data; A feature extraction unit, used for inputting the feature fusion data into the feature extraction layer of the multimodal classification warning model to perform feature extraction for the target disease and obtain a feature extraction result; A classification warning unit is used to input the feature extraction result into the classification layer of the multimodal classification warning model for classification prediction, so as to obtain the symptom warning result of the patient regarding the target symptom.

9. A patient data intelligent analysis device based on CCE data, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the patient data intelligent analysis method based on CCE data as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when called, are used to execute the patient data intelligent analysis method based on CCE data as described in any one of claims 1-7.

Images