Future fetal state prediction method based on multi-modal information fusion sensing

Abstract

A future fetal state prediction method based on multi-modal information fusion sensing, relating to the technical field of medical data processing. The method comprises: acquiring structured maternal health data and unstructured text medical record data of a target user, wherein the maternal health data comprises maternal testing data and maternal feature data; respectively performing data cleaning on the maternal health data and the text medical record data, wherein the text medical record data is converted into structured character medical record data after being subjected to the cleaning; by means of a first sensing model, obtaining first health sensing information on the basis of the maternal health data; by means of a second sensing model, obtaining second health sensing information on the basis of the character medical record data; by means of a fusion sensing model, obtaining multi-modal health sensing information on the basis of the two pieces of health sensing information; and predicting fetal state information on the basis of the multi-modal health sensing information. By means of preset sensing models, fetal health sensing is performed on the basis of multi-modal maternal health information, so that a more accurate and reliable sensing result can be obtained.

Description

A method for predicting the future state of a fetus based on multimodal information fusion perception Technical Field

[0001] This invention relates to the field of medical data processing technology, and in particular to a method for predicting the future state of a fetus based on multimodal information fusion perception. Background Technology

[0002] Regarding the early warning and assessment of premature birth and birth defects, scholars have proposed several evaluation methods: 1) Scoring method: This method first extracts parameters related to fetal heart rate signals and then determines the final fetal status based on the scoring criteria. This alleviates, to some extent, the subjective differences that medical staff may encounter when manually interpreting fetal heart rate and uterine contraction curves and making corresponding clinical decisions. However, current computer-aided diagnostic systems largely rely on simple scoring methods, resulting in low accuracy in assessing fetal health status and a high false positive rate, indicating significant shortcomings. 2) Fetal status classification method based on convolutional neural networks: This method uses one-dimensional fetal heart rate signals as input and compares it with traditional machine learning algorithms. 3) Adaptive neurofuzzy inference system: This system predicts fetal status based on features extracted from fetal heart rate and uterine contraction signals.

[0003] In summary, many scholars have conducted research on premature birth and birth defects, but most of them have adopted traditional clinical research methods. Some scholars have also used deep learning networks for birth defect risk analysis, but the data sources for network learning are limited, the data modalities involved in learning are limited, and the types of risk assessments are limited, resulting in low accuracy of prediction results.

[0004] Therefore, existing technologies still need improvement and development.

[0005] Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for predicting the future state of a fetus based on multimodal information fusion perception, in order to address the above-mentioned deficiencies of the prior art. This method aims to solve the problem that the existing birth defect risk analysis methods based on deep learning networks have limited data sources, modalities, and risk assessment types, resulting in low accuracy of prediction results.

[0007] The technical solution adopted by this invention to solve the problem is as follows:

[0008] In a first aspect, embodiments of the present invention provide a method for predicting the future state of a fetus based on multimodal information fusion perception, the method comprising:

[0009] Acquire structured maternal and fetal health data and unstructured text medical record data of target users, wherein the maternal and fetal health data includes maternal and fetal test data and maternal and fetal characteristic data;

[0010] The maternal and fetal health data and the text medical record data are cleaned separately, wherein the text medical record data is converted into structured character medical record data after cleaning.

[0011] First health perception information is obtained based on the maternal and fetal health data through the first perception model;

[0012] Second health perception information is obtained based on the character medical record data through the second perception model;

[0013] Multimodal health perception information is obtained by fusing the perception model based on the first health perception information and the second health perception information;

[0014] Predict fetal status information based on the multimodal health perception information.

[0015] In one embodiment, the data cleaning method for the text medical record data includes:

[0016] The text medical record data is subjected to text detection, and the text medical record data is asymmetrically segmented based on the text detection results to obtain segmented text;

[0017] The segmented text is subjected to character recognition, and the recognized character sequences are cleaned.

[0018] The character medical record data is obtained based on the cleaned character sequence.

[0019] In one implementation, the step of performing text detection on the text medical record data and performing asymmetric segmentation on the text medical record data based on the text detection results to obtain segmented text includes:

[0020] Based on the text medical record data, predict the boundary points of the text lines and obtain the boundary point coordinate information;

[0021] The text box boundary information is determined based on the boundary point coordinate information;

[0022] The text medical record data is asymmetrically segmented based on the text box boundary information to obtain the segmented text.

[0023] In one embodiment, the step of performing character recognition on the segmented text and cleaning the recognized character sequence includes:

[0024] For each text box in the segmented text, end-to-end recognition is performed on the text sequence of variable length for that text box;

[0025] The identified characters are subjected to semantic, syntactic and logical verification, and useless characters are removed based on the verification results.

[0026] In one implementation, the first perception model is a time-series deep neural network, which includes a recurrent structure and additional layers. Obtaining first health perception information based on the maternal-fetal health data through the first perception model includes:

[0027] The attention weight of each value in the maternal-fetal detection data is calculated based on the data sequence corresponding to the maternal-fetal detection data using the loop structure, thereby obtaining a weighted data sequence.

[0028] The additional layer uses the maternal and fetal feature data as context data, and extracts features from the context data to obtain contextual semantic information.

[0029] The weighted data sequence and the contextual semantic information are fused to obtain a fused feature vector containing attention weights, and the first health perception information is generated based on the fused feature vector.

[0030] In one implementation, the second perception model is a convolutional neural network, and the step of obtaining second health perception information based on the character medical record data through the second perception model includes:

[0031] Feature extraction is performed on the character medical record data, and the extracted features are input into the convolutional neural network;

[0032] The convolutional neural network generates a feature matrix based on the input features, and then inputs the feature matrix into a preset filter to obtain a feature vector.

[0033] The feature vector is input into the attention scoring function to obtain the feature vector containing attention weights;

[0034] The second health perception information is generated based on the feature vector containing attention weights.

[0035] In one implementation, the fusion sensing model is a neural network with embedded common features, and is connected to specific network layers of the first sensing model and the second sensing model. Obtaining multimodal health sensing information through the fusion sensing model based on the first health sensing information and the second health sensing information includes:

[0036] The fused feature vector containing attention weights and the feature vector containing attention weights are weighted and summed to obtain a joint three-dimensional feature vector, wherein the three-dimensional feature vector is used to reflect the abstract features of the corresponding maternal-fetal detection data, the maternal-fetal feature data and the text medical record data;

[0037] The multimodal health perception information is generated based on the three-dimensional feature vector.

[0038] Secondly, embodiments of the present invention also provide a fetal future state prediction device based on multimodal information fusion perception, the device comprising:

[0039] The data acquisition module is used to acquire structured maternal and fetal health data and unstructured text medical record data of the target user, wherein the maternal and fetal health data includes maternal and fetal test data and maternal and fetal characteristic data;

[0040] The data cleaning module is used to clean the maternal and fetal health data and the text medical record data respectively, wherein the text medical record data is converted into structured character medical record data after cleaning.

[0041] The first sensing module is used to obtain first health sensing information based on the maternal and fetal health data through the first sensing model.

[0042] The second sensing module is used to obtain second health sensing information based on the character medical record data through the second sensing model;

[0043] The fusion perception module is used to obtain multimodal health perception information based on the first health perception information and the second health perception information through the fusion perception model;

[0044] The status prediction module is used to predict fetal status information based on the multimodal health perception information.

[0045] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the fetal future state prediction method based on multimodal information fusion perception as described above; the processor is used to execute the programs.

[0046] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions, which are adapted to be loaded and executed by a processor to implement the steps of any of the above-described methods for predicting the future state of a fetus based on multimodal information fusion perception.

[0047] The beneficial effects of this invention are as follows: This invention acquires structured maternal-fetal health data and unstructured text medical record data from a target user. The maternal-fetal health data includes maternal-fetal detection data and maternal-fetal characteristic data. The maternal-fetal health data and the text medical record data are cleaned, and the cleaned text medical record data is transformed into structured character medical record data. A first perception model obtains first health perception information based on the maternal-fetal health data. A second perception model obtains second health perception information based on the character medical record data. A fusion perception model obtains multimodal health perception information based on the first and second health perception information. Fetal status information is predicted based on the multimodal health perception information. This invention, by using a preset perception model and combining multimodal maternal-fetal health information for fetal health perception, can obtain more accurate and reliable perception results. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 is a flowchart illustrating the method for predicting the future state of a fetus based on multimodal information fusion perception provided in an embodiment of the present invention.

[0050] Figure 2 is a logical framework diagram of fetal health perception provided in an embodiment of the present invention.

[0051] Figure 3 is a schematic diagram of the processing flow of text medical record data provided in an embodiment of the present invention.

[0052] Figure 4 is a logical framework diagram of multimodal data perception provided in an embodiment of the present invention.

[0053] Figure 5 is a schematic diagram of the module of the fetal future state prediction device based on multimodal information fusion perception provided in an embodiment of the present invention.

[0054] Figure 6 is a schematic block diagram of the terminal provided in an embodiment of the present invention. Detailed Implementation

[0055] This invention discloses a method for predicting the future state of a fetus based on multimodal information fusion perception. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0056] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] To address the aforementioned deficiencies in existing technologies, this invention provides a method for predicting the future state of a fetus based on multimodal information fusion perception. The method acquires structured maternal-fetal health data and unstructured text medical record data from a target user. The maternal-fetal health data includes maternal-fetal detection data and maternal-fetal feature data. Both the maternal-fetal health data and the text medical record data are cleaned, and the cleaned text medical record data is transformed into structured character medical record data. A first perception model obtains first health perception information based on the maternal-fetal health data. A second perception model obtains second health perception information based on the character medical record data. A fusion perception model obtains multimodal health perception information based on the first and second health perception information. Finally, the fetal state information is predicted based on the multimodal health perception information. This invention, by using a preset perception model and combining multimodal maternal-fetal health information for fetal health perception, can obtain more accurate and reliable perception results.

[0059] As shown in Figure 1, the method includes:

[0060] Step S100: Obtain the target user's structured maternal and fetal health data and unstructured text medical record data, wherein the maternal and fetal health data includes maternal and fetal detection data and maternal and fetal characteristic data.

[0061] Specifically, "maternal-fetal" refers to the pregnant woman (i.e., the target user) and the fetus. Structured data refers to data that can be expressed through data tables. Structured maternal-fetal health data includes two parts. The first part is maternal-fetal testing data, which includes data obtained from examinations and diagnoses conducted by medical institutions, and can be presented in document and / or image modalities. The second part is maternal-fetal characteristic data, which reflects non-health information about the pregnant woman and fetus, such as the pregnant woman's age, place of origin, ethnicity, occupation, genetics, and hobbies; the fetus's gestational age, heart rate, fetal length, and fetal weight; and the types of birth defects in the fetus (nervous system defects, digestive system defects, circulatory system defects, urinary and reproductive system defects, and detailed defect labeling information), which can also be presented in document and / or image modalities. Text medical record data refers to medical record data not recorded in the medical system, including the pregnant woman's text medical record and the fetus's text medical record. This medical record data records health information in paper image format and serves as a supplement to maternal-fetal health information.

[0062] Step S200: Perform data cleaning on the maternal and fetal health data and the text medical record data respectively, wherein the text medical record data is converted into structured character medical record data after cleaning.

[0063] To meet the needs of big data network learning, it is essential to automatically clean the large volume of disorganized maternal-fetal testing and characteristic data from various sources, removing useless and incorrect data to obtain complete, accurate, and standardized formatted real data. Simultaneously, feature extraction, semantic recognition, and text character conversion are required for the medical record images of pregnant women and fetuses to obtain characterized text medical record diagnostic content and results—i.e., character medical record data—thus achieving text medical record data cleaning. For structured maternal-fetal health data: This data exists in structured two-dimensional tables. The goal of cleaning is to remove the disorganized, incomplete, and irregular maternal-fetal health data from different sources and map it to a unified, complete, and standardized structured three-dimensional dataset. For text medical record data: Text medical records contain disease content, disease analysis, and diagnostic conclusions. Different doctors, different medical record sources, different descriptions of symptoms, and different recording methods lead to the complexity and diversity of text medical records. To understand the disease information reflected in text medical records and meet the needs of deep learning in computational networks, it is necessary to perform semantic understanding, content filtering, boundary planning, and character recognition on the text medical records. This process transforms health information expressed in image form into text information represented in character form. Through deep fusion perception and iterative cleaning of the text characters, data information that can accurately reflect maternal and fetal health can be obtained.

[0064] In one implementation, as shown in Figure 2, the method for cleaning structured maternal-fetal health data includes three stages: data integration, outlier handling, and missing value handling (outliers and missing values ​​are both abnormal values). Data integration includes field extraction, data table normalization, and deduplication. For field extraction, entity recognition and attribute redundancy must be considered. The purpose of entity recognition is to select corresponding fields and merge their corresponding data to obtain the data fields required for a standard structured data table. For data table normalization, this involves forming a standardized data table from data tables of different sources and formats through field extraction. The standardized data table is a pre-constructed normalized data table, and its construction method is determined based on the fetal health prediction task / premature birth and birth defect risk prediction task, while also considering algorithm efficiency. For deduplication, after field filtering and extraction and multi-table normalization integration, duplicate data needs to be removed. Outlier handling mainly addresses data deviations from normal values ​​due to errors in information storage and transformation, or the occurrence of certain complex conditions. Outlier handling primarily employs correction or removal methods. Missing value handling mainly uses deletion or imputation methods.

[0065] In one implementation, methods for removing duplicate data include:

[0066] Based on the standardized data table, data clustering is performed to obtain several data categories;

[0067] For each type of data, calculate the similarity of the data in that type, and determine whether there is duplicate data based on the similarity. If there is duplicate data, remove the duplicate data.

[0068] After removing duplicate data, adjust the search radius and continue performing the data clustering steps based on the standardized data table until there are no duplicate data in each category.

[0069] Specifically, the removal of duplicate data incorporates a clustering algorithm. First, the standardized data table is clustered. Then, the similarity of data within the same cluster is calculated to determine if duplicates exist. If duplicates are found, only one data point is retained, and the others are removed. The search radius is then adjusted to modify the density parameters of each cluster, and this process is iterated and repeated until each cluster contains approximately identical data points, thus completing the detection and removal of duplicate data.

[0070] In one implementation, outlier detection methods include:

[0071] The data is clustered based on the remaining data after removing duplicate data using a preset clustering algorithm, resulting in several data clusters.

[0072] By using the distance information of each data cluster, it is determined whether there are global outliers;

[0073] The presence of local outliers is determined by using the density information of each data cluster.

[0074] Specifically, outliers can be categorized into global outliers and local outliers. Global outliers are detected using distance-based methods, while local outliers are detected using density-based methods. Distance-based and density-based outlier detection methods require measuring the Euclidean distance and outlier factor of each object relative to other objects, leading to unimaginable time consumption as the data scale increases. Clustering algorithms can use unsupervised learning to cluster objects with similar characteristics into multiple data clusters of varying sizes. This embodiment combines clustering algorithms, distance-based outlier detection, and density-based outlier detection to find outliers, thereby reducing the problem scale and lowering the algorithm's time complexity.

[0075] In one implementation, the methods for handling missing values ​​include:

[0076] Use the data corresponding to the missing values ​​as the data to be imputed;

[0077] The data to be imputed is iteratively sampled to obtain a posterior distribution;

[0078] The missing values ​​are filled by extracting values ​​from the posterior distribution.

[0079] Specifically, the maternal-fetal testing data is primarily numerical, therefore imputation is used to handle missing data. Missing values ​​can be numerical, categorical, or count values. This embodiment uses an iterative approach with a multiple regression model to imput missing values. The data corresponding to the missing values ​​is taken as the data to be imputed. First, iterative sampling is performed on the data to be imputed to obtain a posterior distribution, and values ​​are drawn from this posterior distribution to fill in the missing values. The imputation method provided in this embodiment has a faster convergence speed and does not require normalizing the dataset to a normal distribution.

[0080] In another implementation, the maternal-fetal detection data is time-series data, and the method for handling missing values ​​includes:

[0081] Use time series data containing missing values ​​as the data to be imputed;

[0082] The missing values ​​are imputed by using the self-attention mechanism through the time step correlation in the time series data, where the time step correlation refers to the correlation between observations at different time points within each variable.

[0083] Furthermore, in this embodiment, an imputation module is pre-built based on a self-attention mechanism to perform the above-mentioned step of imputing missing values ​​by utilizing the self-attention mechanism through the correlation of time steps in the time series data. The training method of the imputation module includes:

[0084] Perform time alignment on the input time series samples; (the time series can be irregular time series);

[0085] Randomly remove a predetermined proportion of observations to obtain an indicator mask I that records the positions of the removed values, a missing mask M that records the missing patterns, and the resulting time series data.

[0086] The time series data obtained after removing some observations The missing mask M is input together with the data into the interpolation module, and the time series data is obtained after interpolation.

[0087] Based on time series data The loss value of the interpolation module is calculated using the indicator mask I, and the parameters of the interpolation module are adjusted based on the loss value to train the interpolation module.

[0088] Specifically, the interpolation module based on the self-attention mechanism can perform time series interpolation by utilizing the time-step correlations of variables in time series data. (Time series data) Along with the missing mask M, it is input into the imputation module, which enables the model to capture the distribution of observations to a certain extent, helping the model understand the time series.

[0089] In one implementation, the data cleaning method for the text medical record data includes:

[0090] The text medical record data is subjected to text detection, and the text medical record data is asymmetrically segmented based on the text detection results to obtain segmented text;

[0091] The segmented text is subjected to character recognition, and the recognized character sequences are cleaned.

[0092] The character medical record data is obtained based on the cleaned character sequence.

[0093] To achieve a true understanding of text-based medical records and achieve deep integration of text-based medical records and detection data, this embodiment requires the conversion from text-based medical record data to text character information. The medical record content, represented in image form, is converted into a character-based representation based on the understanding and perception of the content. Then, through iterative convergence cognitive calculations on the complex and disorganized character content, redundant, erroneous, and irregular medical record content is cleaned to generate a complete, accurate, and standardized character-based medical record data. Specifically, as shown in Figure 3, text-based medical record data cleaning can be divided into four stages: First, the text-based medical record data is preprocessed; then, text detection and recognition are performed on the image-based text-based medical record files. Text detection and extraction require understanding the text-based medical record content and semantics. This is achieved by performing asymmetric segmentation on the medical record image, and then extracting the text content from the segmented content to obtain segmented text; finally, text characters are extracted from the segmented text, and the extracted text character sequences are filtered or cleaned to obtain usable character sequences.

[0094] In one implementation, the preprocessing method for text medical record data includes: text medical record scanning and text image medical record correction. Text medical record scanning can be processed using optical character recognition (OCR) technology, and text image correction includes image binarization, tilt correction, white edge removal, and text curvature correction.

[0095] In one implementation, the step of performing text detection on the text medical record data and performing asymmetric segmentation on the text medical record data based on the text detection results to obtain segmented text includes:

[0096] Based on the text medical record data, predict the boundary points of the text lines and obtain the boundary point coordinate information;

[0097] The text box boundary information is determined based on the boundary point coordinate information;

[0098] The text medical record data is asymmetrically segmented based on the text box boundary information to obtain the segmented text.

[0099] Specifically, to facilitate text detection and segmentation, this embodiment can pre-build a scene text detection model, predict the boundary points of text lines using the scene text detection model, and then perform a weighted average of the coordinates of the boundary points to determine the text box boundaries, thereby achieving image segmentation based on text boxes.

[0100] In one implementation, the step of performing character recognition on the segmented text and cleaning the recognized character sequence includes:

[0101] For each text box in the segmented text, end-to-end recognition is performed on the text sequence of variable length for that text box;

[0102] The identified characters are subjected to semantic, syntactic and logical verification, and useless characters are removed based on the verification results.

[0103] Specifically, to facilitate character recognition and cleaning, this embodiment pre-constructs a convolutional recurrent neural network (RNN), which can be used in conjunction with the aforementioned scene text detection model to construct a hybrid algorithm model. The RNN is used for character recognition of the text box content. Specifically, it performs end-to-end recognition of the variable-length text sequence within the text box to obtain a sequence of text information reflecting the maternal and fetal medical records, i.e., a medical record character sequence. Text character cleaning primarily involves further refining the medical record character sequence after obtaining it through semantic, syntactic, and logical understanding. This removes useless information and extracts relevant information such as maternal and fetal disease types, detection data, and detection conclusions, constructing a text character sequence expressed in a standard language and format to obtain character-based medical record data. In one implementation, this embodiment can employ natural language processing technology to implement the above character cleaning process.

[0104] In one implementation, obtaining the character medical record data based on the cleaned character sequence includes:

[0105] The cleaned character sequences are mapped to a three-dimensional dataset in text format as supplementary character medical record data. The three-dimensional dataset consists of three dimensions of data: maternal-fetal detection data, maternal-fetal feature data, and character medical record data.

[0106] Specifically, multi-source, multi-modal, and multi-type maternal and fetal detection data, feature data, and medical record data, including structured detection data and unstructured text medical record data, as well as single feature data and time-series recorded continuous medical record data, can construct multi-source, multi-dimensional, multi-modal, and multi-structured maternal and fetal health data relationships, thereby improving the accuracy of subsequent fetal status prediction.

[0107] Step S300: Obtain first health perception information based on the maternal and fetal health data through the first perception model.

[0108] Specifically, maternal and fetal health data includes maternal and fetal detection data and maternal and fetal characteristic data. In this embodiment, a first perception model is pre-constructed. The first perception model uses historical maternal and fetal health data as training data and fetal characteristic information from it as labeled data to predict fetal health status. In other words, fetal characteristic information is not only training data but also the output training result, and it is also automatically labeled information for maternal and fetal health training data. After iterative training with a large amount of training data, a relationship model for realizing the perception of fetal status and maternal and fetal health data is obtained, namely the first perception model.

[0109] For example, predictive information for the future state of the fetus includes predictions of preterm birth and birth defects. Maternal-fetal test data obtained through medical examinations are not consistent in most cases; for example, pulse rate and blood routine tests may show different proportions of preterm birth and birth defects. Using maternal-fetal health data as predictive data for preterm birth and birth defects, and using preterm birth and birth defects as labeled data, a first-order perception model is constructed and trained.

[0110] In one implementation, the first perception model is a time-series deep neural network, which includes a recurrent structure and additional layers. Obtaining first health perception information based on the maternal-fetal health data through the first perception model includes:

[0111] The attention weight of each value in the maternal-fetal detection data is calculated based on the data sequence corresponding to the maternal-fetal detection data using the loop structure, thereby obtaining a weighted data sequence.

[0112] The additional layer uses the maternal and fetal feature data as context data, and extracts features from the context data to obtain contextual semantic information.

[0113] The weighted data sequence and the contextual semantic information are fused to obtain a fused feature vector containing attention weights, and the first health perception information is generated based on the fused feature vector.

[0114] Specifically, this embodiment uses a time-series deep neural network to construct the first perception model. In practical applications, maternal and fetal health data is input into the time-series deep neural network. The recurrent structure of the time-series deep neural network processes the data sequence corresponding to the maternal and fetal detection data. Specifically, it predicts each value in the fused feature vector based on the historical values ​​of the maternal and fetal detection data, concatenates the memory layer of the previous time step with the memory layer of the current time step, and then performs a Hadamard product with the probability values ​​obtained by inputting the source data at the same time step into the softmax function to obtain the attention weights. The maternal and fetal feature data is used as context data through an additional layer of the time-series deep neural network, and the features of the context data are extracted to obtain contextual semantic information. The data sequence and contextual semantic information are fused to obtain a fused feature vector with joint representation. Finally, the first health perception information is generated based on the fused feature vector containing attention weights.

[0115] For example, time-series deep neural networks can use networks such as LSTM and GRU. Individual differences can affect the final result judgment, so the contextual semantic information of different targets needs to be optimized. Based on the data, a recurrent structure is applied to process the variable-length sequence of historical measurements, and the contextual semantic information is fused with this sequence. This is mainly achieved by introducing an additional layer to extract features from the contextual data. To better predict the future value of the target, these two sequences are fused, with the aim of adjusting the network level through maternal feature vector factors. Specifically, firstly, the result is compressed through a fully connected layer to obtain the processed contextual semantic weights, which are then fused with the processed sequence data of the source data to obtain the joint representation of the final target conditions. Secondly, each value is predicted based on previous detection data. The memory layer of the predicted data from the previous time step is concatenated with the memory layer of the current time step, and then the probability is obtained by performing a Hadamard product with the source data at the same time step through a Softmax function to obtain the attention weight coefficients. Finally, learning is performed based on different data to achieve the accuracy of each weight coefficient. This generates structured maternal-fetal health monitoring information obtained by fusing detection data and feature data, performs result perception, and finally obtains a relationship model between fetal preterm birth and birth defects and maternal-fetal detection data perception.

[0116] In one implementation, a hidden layer is added to the source data before the historical data is output to process the context data, and an activation function is used to avoid data loss or gradient vanishing problems.

[0117] Step S400: Obtain second health perception information based on the character medical record data through the second perception model.

[0118] Specifically, the perception of textual character information differs from the aforementioned perception of structured maternal and fetal health information. Textual character information is not discrete records or field data, but rather continuous character data without a uniform format or order. Each medical record contains different disease information and has a different recording format. Furthermore, due to doctors' professional habits, the analysis and description of diseases vary, thus constituting the diversity and complexity of character-based medical record data. To facilitate the processing of character-based medical record data, this embodiment pre-builds and trains another deep learning model, namely the second perception model.

[0119] For example, this embodiment addresses the complexity and diversity of character-based medical record data by using premature birth and birth defects as labeled data for character-based medical record data (i.e., text-based health information). It mines explicit and implicit health information from the text characters to construct a second perception model suitable for single or multiple text-based records in a data table. Through this second perception model, the impact of merging medical records and character region fusion perception on perception quality and efficiency under multiple text-based records can be explored, achieving true perception of text-based medical records.

[0120] In one implementation, the second perception model is a convolutional neural network, and obtaining the second health perception information based on the character medical record data through the second perception model includes:

[0121] Feature extraction is performed on the character medical record data, and the extracted features are input into the convolutional neural network;

[0122] The convolutional neural network generates a feature matrix based on the input features, and then inputs the feature matrix into a preset filter to obtain a feature vector.

[0123] The feature vector is input into the attention scoring function to obtain the feature vector containing attention weights;

[0124] The second health perception information is generated based on the feature vector containing attention weights.

[0125] Specifically, for the converted character medical record data, feature extraction is first performed, and then a convolutional neural network is used to process the input features, with each word represented as a three-dimensional vector. A fixed number of features are selected, and two elements before and after each feature are chosen as row vectors and padded with zeros to expand the number of elements. The resulting feature matrix is ​​then subjected to convolution, max pooling, and other operations according to the selected filter size to obtain a fixed-length feature vector of the text. In addition, an attention scoring function is added to the embedding vector, and attention weights are generated after activation to increase the local attention and importance of the network, thereby obtaining second health perception information that reflects the health information of the maternal-fetal text medical record. This yields a relationship model between fetal preterm birth and the perception of maternal-fetal text medical record data, i.e., the second perception model.

[0126] Step S500: Obtain multimodal health perception information based on the first health perception information and the second health perception information through the fusion perception model.

[0127] Specifically, the two-dimensional structured maternal-fetal detection information and maternal-fetal characteristic information, together with the two-dimensional character medical record data, constitute a complete three-dimensional data space for maternal-fetal health information. Because the detection information and character medical record data differ significantly in storage structure, expression format, and data relationships, resulting in differences in their content and expression methods, this embodiment also constructs a fusion perception model to fuse the first and second health perception information to obtain multimodal health perception information.

[0128] For example, maternal-fetal examination data might be recorded as "systolic blood pressure = 130, diastolic blood pressure = 90," while text-based medical record data might be represented as "systolic blood pressure: 130 mmHg, diastolic blood pressure: 90 mmHg." These semantic and grammatical differences create discrepancies in health information perception. Based on the semantic and grammatical characteristics of maternal-fetal examination data and text-based medical record data, a perceptual training network suitable for both data types can be designed to study their semantic and grammatical relationship. Furthermore, a common code table can be designed for both, constructing a fusion perceptual model of the examination data network and the text-based medical record data network. This enables comprehensive health information perception within a three-dimensional health data space encompassing maternal-fetal examination data, maternal-fetal feature data, and text-based medical record data. This ensures that, regardless of whether maternal-fetal examination data and text-based medical record data are equivalent or asymmetrical, and under multi-source, multi-dimensional, multi-modal, and multi-type data conditions, multiple heterogeneous networks can collaborate to obtain a unified, standardized, and reproducibly computable fetal health perception model.

[0129] In one implementation, the fusion sensing model is a neural network with embedded common features, and is connected to specific network layers of the first sensing model and the second sensing model. Obtaining multimodal health sensing information through the fusion sensing model based on the first health sensing information and the second health sensing information includes:

[0130] The fused feature vector containing attention weights and the feature vector containing attention weights are weighted and summed to obtain a joint three-dimensional feature vector, wherein the three-dimensional feature vector is used to reflect the abstract features of the corresponding maternal-fetal detection data, the maternal-fetal feature data and the text medical record data;

[0131] The multimodal health perception information is generated based on the three-dimensional feature vector.

[0132] Specifically, the fusion of maternal and fetal multimodal health information perception is built upon the first two perception steps. To achieve fused perception, this embodiment employs a neural network structure with embedded common features. This structure includes a joint representation, which has a central network connecting specific neural network layers of different modalities. First, the relationship between fetal preterm birth and maternal and fetal detection data perception, as well as maternal and fetal text medical record data perception, needs to be obtained separately. Simultaneously, the feature vectors and attention weight coefficients of the structured fusion data are obtained. These two are then weighted and summed with the previous layers to obtain a joint representation, thereby achieving the goal of combining the implicit features of different modalities. Each joint representation vector is a three-dimensional vector, including abstract features of maternal and fetal detection data, text medical record data, and feature data. The joint representation serves as a hidden layer of the neural network.

[0133] In one implementation, a combination of early fusion (data level) and late fusion (decision level) is used to obtain a joint representation, which is then output to the classifier for further perceptual prediction processing (as shown in Figure 4). The fusion training process involves matching the pregnant woman's feature records with the corresponding fetal feature records, using this as an index to link the detection data and medical record data, and continuously iterating to train the parameters of the hidden layer.

[0134] Step S600: Predict fetal status information based on the multimodal health perception information.

[0135] Specifically, the predicted fetal status information can reflect fetal health risks, such as the risk of premature birth and / or birth defects; that is, the predicted fetal status information can represent the future state of the fetus. The goal of this embodiment is to find the pattern between fetal health risks and maternal and fetal health, thereby obtaining accurate fetal health risk perception. Text medical record data, as a supplement to maternal and fetal health information, can form a more complete fetal health dataset. Therefore, this embodiment uses multimodal health perception information generated jointly from maternal and fetal examination information, maternal and fetal characteristic information, and text medical record data to predict the future state of the fetus. Due to the richness of health data, the predicted fetal status information will be more accurate, leading to a more refined fetal health risk assessment and predictive warning.

[0136] Based on the above embodiments, the present invention also provides a fetal future state prediction device based on multimodal information fusion perception, as shown in Figure 5, the device comprising:

[0137] The data acquisition module 01 is used to acquire structured maternal and fetal health data and unstructured text medical record data of the target user, wherein the maternal and fetal health data includes maternal and fetal detection data and maternal and fetal characteristic data;

[0138] The data cleaning module 02 is used to clean the maternal and fetal health data and the text medical record data respectively, wherein the text medical record data is converted into structured character medical record data after cleaning.

[0139] The first sensing module 03 is used to obtain first health sensing information based on the maternal and fetal health data through the first sensing model;

[0140] The second sensing module 04 is used to obtain second health sensing information based on the character medical record data through the second sensing model;

[0141] The fusion perception module 05 is used to obtain multimodal health perception information based on the first health perception information and the second health perception information through the fusion perception model;

[0142] The state prediction module 06 is used to predict fetal state information based on the multimodal health perception information.

[0143] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which is shown in Figure 6. The terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for predicting the future state of the fetus based on multimodal information fusion perception. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0144] Those skilled in the art will understand that the principle block diagram shown in Figure 6 is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs include instructions for performing a method for predicting the future state of a fetus based on multimodal information fusion perception.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0147] In summary, this invention discloses a method for predicting the future state of a fetus based on multimodal information fusion perception. The method acquires structured maternal-fetal health data and unstructured text medical record data from a target user. The maternal-fetal health data includes maternal-fetal detection data and maternal-fetal feature data. The maternal-fetal health data and the text medical record data are cleaned, and the cleaned text medical record data is transformed into structured character medical record data. A first perception model obtains first health perception information based on the maternal-fetal health data. A second perception model obtains second health perception information based on the character medical record data. A fusion perception model obtains multimodal health perception information based on the first and second health perception information. The fetal state information is predicted based on the multimodal health perception information. This invention, by using a preset perception model and combining multimodal maternal-fetal health information for fetal health perception, can obtain more accurate and reliable perception results.

[0148] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for predicting the future state of a fetus based on multi-modal information fusion perception, characterized in that, The method comprises: obtaining structured maternal and fetal health data and unstructured text medical record data of a target user, wherein the maternal and fetal health data comprises maternal and fetal detection data and maternal and fetal feature data; respectively performing data cleaning on the maternal and fetal health data and the text medical record data, wherein the text medical record data is converted into structured character medical record data after being cleaned; obtaining first health perception information according to the maternal and fetal health data through a first perception model; obtaining second health perception information according to the character medical record data through a second perception model; obtaining multi-modal health perception information according to the first health perception information and the second health perception information through a fusion perception model; predicting fetal state information according to the multi-modal health perception information.

2. The method for predicting the future state of the fetus based on multi-modal information fusion perception according to claim 1, characterized in that, The data cleaning method of the text medical record data comprises: performing text detection on the text medical record data, and performing asymmetric segmentation on the text medical record data according to a text detection result to obtain segmented text; performing character recognition on the segmented text, and cleaning a recognized character sequence; obtaining the character medical record data according to the cleaned character sequence.

3. The method for predicting the future state of the fetus based on multi-modal information fusion perception according to claim 2, characterized in that, The method of performing text detection on the text medical record data and performing asymmetric segmentation on the text medical record data according to a text detection result to obtain segmented text comprises: predicting boundary points of text lines according to the text medical record data to obtain boundary point coordinate information; determining text box boundary information according to the boundary point coordinate information; performing asymmetric segmentation on the text medical record data according to the text box boundary information to obtain the segmented text.

4. The method for predicting the future state of the fetus based on multi-modal information fusion perception according to claim 2, characterized in that, The method of performing character recognition on the segmented text and cleaning a recognized character sequence comprises: performing end-to-end recognition on an indefinite-length text sequence of each text box in the segmented text; performing semantic, grammatical and logical verification on the recognized characters, and cleaning away useless characters according to a verification result.

5. The method for predicting the future state of the fetus based on multi-modal information fusion perception according to claim 1, characterized in that, The first perception model is a time series deep neural network, the time series deep neural network comprises a loop structure and an additional layer, and the method of obtaining first health perception information according to the maternal and fetal health data through the first perception model comprises: calculating attention weights of each value in the maternal and fetal detection data to obtain a weight data sequence through the loop structure according to a data sequence corresponding to the maternal and fetal detection data; extracting features of context data from the maternal and fetal feature data as the context data through the additional layer to obtain context semantic information; fusing the weight data sequence and the context semantic information to obtain a fusion feature vector containing attention weights, and generating the first health perception information according to the fusion feature vector.

6. The method for predicting the future state of the fetus based on multi-modal information fusion perception according to claim 5, characterized in that, The second perception model is a convolutional neural network, and the method of obtaining second health perception information according to the character medical record data through the second perception model comprises: extracting features from the character medical record data, and inputting the extracted features into the convolutional neural network; generating a feature matrix through the convolutional neural network according to the input features, and inputting the feature matrix into a preset filter to obtain a feature vector; inputting the feature vector into an attention score function to obtain the feature vector containing attention weights; generating the second health perception information according to the feature vector containing attention weights.

7. The method for predicting the future state of the fetus based on multi-modal information fusion perception according to claim 6, characterized in that, The fusion perception model is a neural network with embedded common features and is connected with specific network layers of the first perception model and the second perception model, and the multi-modal health perception information is obtained according to the first health perception information and the second health perception information through the fusion perception model, including: performing weighted summation according to the fusion feature vector containing attention weights and the feature vector containing attention weights to obtain a three-dimensional feature vector of joint representation, wherein the three-dimensional feature vector is used to reflect the abstract features of the maternal-fetal detection data, the maternal-fetal feature data and the text medical record data having a corresponding relationship; generating the multi-modal health perception information according to the three-dimensional feature vector.

8. A device for predicting the future state of a fetus based on multi-modal information fusion perception, characterized in that, The device comprises: a data acquisition module configured to acquire structured maternal-fetal health data and unstructured text medical record data of a target user, wherein the maternal-fetal health data comprises maternal-fetal detection data and maternal-fetal feature data; a data cleaning module configured to clean the maternal-fetal health data and the text medical record data respectively, wherein the text medical record data is converted into structured character medical record data after cleaning; a first perception module configured to obtain first health perception information according to the maternal-fetal health data through a first perception model; a second perception module configured to obtain second health perception information according to the character medical record data through a second perception model; a fusion perception module configured to obtain multi-modal health perception information according to the first health perception information and the second health perception information through a fusion perception model; a state prediction module configured to predict fetal state information according to the multi-modal health perception information.

9. A terminal, characterized by comprising: The terminal comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the fetal future state prediction method based on multi-modal information fusion perception according to any one of claims 1-7; the processor is used to execute the program.

10. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are suitable for being loaded and executed by the processor to implement the steps of the fetal future state prediction method based on multi-modal information fusion perception according to any one of claims 1-7. The instructions are suitable for being loaded and executed by the processor to implement the steps of the fetal future state prediction method based on multi-modal information fusion perception according to any one of claims 1-7.

Images