A multi-modal medical information fusion fetal auxiliary diagnosis system

Abstract

The application discloses a multi-modal medical information fusion fetal auxiliary diagnosis system, which comprises a signal preprocessing module, an image generation module, a text extraction module, a multi-modal fusion module and an output module; wherein the multi-modal fusion module utilizes an image-text feature fusion network ITFN to perform feature extraction on image data acquired by the image generation module and text data acquired by the text extraction module, to obtain an image feature vector M i and a text feature vector M t ; the image feature vector M i and the text feature vector M t are weighted and fused to obtain a multi-modal feature vector Z f ; and the output module uses a full connection layer to reduce dimensions of the multi-modal feature vector Z f , and outputs a result of normal and pathological sample classification. The application adopts different models to learn local semantics and acquire more relevant knowledge information, realizes effective fetal distress pathological diagnosis and reduces a misdiagnosis rate, and has wide practical application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical health, in particular to the monitoring technology of pregnant women and fetuses, and specifically to a multi-modal medical information fusion fetal auxiliary diagnosis system. BACKGROUND

[0002] With the development of economy, the number of elderly pregnant women has increased sharply. Fetal acidosis caused by hypoxia can cause irreversible consequences such as organ damage and even death. Therefore, it is crucial to ensure the health of pregnant women and fetuses, which is a key research topic and breakthrough direction in the field of international medical health today, and has extremely important significance.

[0003] Fetuses in the third trimester of pregnancy may experience hypoxia in the uterus, leading to acidosis, which can cause irreversible consequences such as fetal organ damage and even death. Therefore, it is crucial to ensure the health of pregnant women and fetuses. Currently, Cardiotocography (CTG) technology is a commonly used method for obstetricians to assess whether the fetus is in distress and whether emergency intervention is needed. CTG technology assesses fetal status by continuously and synchronously recording Fetal Heart Rate (FHR) and Uterine Contraction (UC) signals. However, due to the dynamic and complex nature of CTG signals, there are significant differences in signal interpretation among different doctors, which can lead to unnecessary cesarean sections. In order to reduce the misdiagnosis rate, researchers have developed Intelligent Cardiotocography (ICTG) methods based on machine learning and deep learning tools, combined with guidelines from the International Federation of Gynecology and Obstetrics, to intelligently assist in diagnosing fetal distress. With the development of computer technology, ICTG methods are also constantly updated, from One-Dimensional (1D) signal processing to Two-Dimensional (2D) picture-based feature extraction, from traditional machine learning methods to deep learning methods, and from single-modal feature extraction to multi-modal feature consideration. However, the existing technical challenges mainly lie in the following aspects:

[0004] (1) ICTG based on 1D signal mainly relies on mathematical analysis and pattern recognition of FHR signals, although it achieves a certain degree of intelligence, but has limitations in capturing the complex nonlinear relationships and potential patterns of sequential signals.

[0005] (2) Based on the combination of 2D images and deep learning, feature representations are learned from the raw data, achieving better performance than traditional machine learning algorithms. However, ICTG based solely on FHR is limited to specific information sources and cannot fully reflect the physiological condition of the fetus; moreover, in clinical practice, obstetricians will combine real-world factors such as maternal age and gestational age to comprehensively analyze the fetal condition.

[0006] Therefore, a new auxiliary diagnostic system is urgently needed. Summary of the Invention

[0007] The first objective of this invention is to address the aforementioned technical challenges in the practical application of fetal assisted diagnostic technology and to provide a fetal assisted diagnostic system that integrates multimodal medical information. This system not only considers different data modalities but also extracts features based on the characteristics of different models, thereby achieving effective pathological diagnosis of fetal distress and assisting clinicians in monitoring distressed fetuses.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A multimodal medical information fusion-based fetal assisted diagnostic system includes:

[0010] The signal preprocessing module removes noise from the original FHR signal to obtain a clean FHR signal;

[0011] The image generation module converts the clean FHR signal into image data;

[0012] The text extraction module extracts the parametric features of the clean FHR signal and combines them with clinical physiological information to obtain text data;

[0013] The multimodal fusion module utilizes the Image-Text Feature Fusion Network (ITFN) to extract features from image and text data, obtaining the image feature vector M. i and text feature vector M t ; to transform the image feature vector M i and text feature vector M t Weighted fusion is performed to obtain the multimodal feature vector Z f The image-text feature fusion network ITFN includes an image encoder and a text encoder. The image encoder uses a ViT network, and the text encoder uses a convolutional neural network (CNN).

[0014] The output module uses fully connected layers to process the multimodal feature vector Z. f Dimensionality reduction, outputting the classification results of normal and pathological samples.

[0015] Furthermore, the image generation module employs a Markov Transition Field (MTF).

[0016] Furthermore, the parameter features include morphological, time-domain, and frequency-domain parameters, and the clinical physiological information includes maternal age and gestational age.

[0017] Furthermore, the ViT network comprises the following sequentially connected components:

[0018] The Patch Embedding module divides the input image into 16x16 patches, flattens each patch, and converts it into a one-dimensional vector representation.

[0019] Position Embedding module: Adds position encoding to each patch to preserve the spatial position information of the image, ensuring that the Transformer can utilize positional relationships;

[0020] The encoder group comprises 12 encoder blocks, each containing a multi-head attention module and a multilayer perceptron (MLP) block. Each encoder block consists of a first-level Layer Norm layer, a multi-head attention layer, and a first-level Dropout layer connected in sequence. The input of the encoder block is added to the output of the first-level Dropout layer and then fed into the second-level Layer Norm layer. The second-level Layer Norm layer is sequentially connected to the MLP linear layer and the second-level Dropout layer. The input of the second-level Layer Norm layer is added to the output of the second-level Dropout layer and then fed into the output of each encoder block. The multilayer perceptron (MLP) block consists of two fully connected layers, with a nonlinear transformation using the GELU activation function in between, and includes path drop (DropPath) to prevent overfitting.

[0021] Classification Head (MLP Head): The classification head includes a pre-logits layer and a linear layer; the output of the encoder group is fed into the classification head after passing through a Layer Norm layer, and the linear layer outputs the image feature vector M. i .

[0022] Furthermore, the convolutional neural network (CNN) includes a first convolutional layer, a second convolutional layer, a max-pooling layer, and a fully connected layer connected in sequence, with the fully connected layer outputting a text feature vector M. t The output of the max pooling layer is flattened by Flatten and then input into the fully connected layer.

[0023] Furthermore, the weighted fusion specifically refers to:

[0024] Zf =gM i +(1-g)M t (4)

[0025]

[0026] Where g represents the weights, W1 and W2 are the learnable weight matrices of two different linear layers; σ(·) represents the activation function; Z f This represents a multimodal feature vector.

[0027] A second object of the present invention is to provide an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the system described above.

[0028] A third objective of the present invention is to provide a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the aforementioned system.

[0029] The specific modules of this invention are as follows:

[0030] Firstly, in order to address the problem of insufficient fetal pathology data, this invention proposes a method for generating images using existing datasets to compensate for the lack of data, thereby improving the generalization ability and robustness of the model.

[0031] Secondly, this invention proposes a multimodal data fusion fetal diagnostic model (Multi-FHRNet). Considering real-world clinical factors, it integrates three types of features (morphological, temporal, and frequency domain) of maternal age, gestational age, and FHR signal as text data, further improving model performance. By effectively utilizing various modal information, experimental results demonstrate that the method of this invention achieves significant performance in determining whether the fetus is hypoxic.

[0032] Thirdly, this invention designs an Image-Text Fusion Net (ITFN), in which a Vision Transformer (ViT) and a Convolutional Neural Network (CNN) serve as the image encoder and text encoder, respectively, to extract features of different modalities. Furthermore, it implements Adaptive Weighting (ADW) of feature vectors of different modalities instead of direct concatenation, effectively utilizing various modal information.

[0033] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0034] The multimodal medical information fusion method provided in this invention aims to improve the accuracy and efficiency of fetal pathology diagnosis through innovative technology:

[0035] 1. This invention constructs multimodal data based on image data and clinical information, integrating clinically relevant factors to make diagnostic results more consistent with actual medical needs. This construction of multimodal data not only improves diagnostic accuracy but also enhances the model's generalization ability, enabling it to be better applied in different clinical scenarios.

[0036] 2. In this invention, ViT and Convolutional Neural Networks (CNNs) process data from different modalities, enabling more efficient processing of both local and global features of different modalities. ViT's attention mechanism better captures relationships between images, while CNN excels at capturing local features; the combination of the two provides a more comprehensive analysis of multimodal data. Simultaneously, an adaptive weighting method is used instead of direct feature vector concatenation. By dynamically selecting and integrating feature information from different sources, model performance is significantly improved, and the utilization of multimodal information is optimized. Through these methods, this invention not only improves the accuracy of fetal pathology diagnosis but also enhances the model's generalization ability and clinical application value. These innovations demonstrate the advanced nature and practicality of this invention in the field of multimodal medical information fusion. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This invention provides a multimodal information fusion fetal auxiliary diagnostic architecture.

[0039] Figure 2 This is the effect of converting a 1D signal into a 2D image used in this invention;

[0040] Figure 3 The ViT-based image data encoder provided for this invention;

[0041] Figure 4 The CNN-based text modality encoder network structure provided by this invention; Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The fetal assisted diagnostic system based on multimodal medical information fusion has great practical application value and can provide guidance for existing medical and health fields on how to monitor whether the fetus is in distress, hypoxia, and whether human intervention is needed.

[0044] Figure 1 The multimodal medical information fusion architecture provided by this invention includes:

[0045] (1) Data preprocessing

[0046] Noise Reduction: FHR signals are typically acquired clinically using Doppler ultrasound probes or fetal scalp electrodes, and are susceptible to noise during acquisition. Fetal or maternal movement, sensor malfunction, and other phenomena can cause signal loss. Noise in FHR signals usually manifests as artifacts (or spikes) and missing values ​​(periods when FHR is zero). Therefore, the following three steps are used to reduce these two types of noise: First, when the FHR signal value is 0 and the duration is greater than 15 seconds, it is directly removed; otherwise, linear interpolation is performed. Second, when the FHR signal is unstable, i.e., the absolute value of two adjacent points is greater than 25 bpm (|x(i)-x(i-1)|≥25), interpolation is performed between the initial sampling point and the first point of the next stable part. Finally, when the FHR value is greater than 200 bpm (x(i)≥200) or less than 50 bpm (x(i)≤50), cubic spline interpolation is used for filling.

[0047] Converting to Image: This invention achieves the conversion of 1D signal data to 2D image data through MTF. First, the FHR is divided into Q quantile intervals according to its value range; second, a Q×Q Markov transition matrix W is established, as shown in equation (1), w ij Let M represent the transfer frequency from quantile bin i to j. Finally, construct the Markov transfer field M, as shown in equation (2), M ij Represented as data q at any two time points i ,q j The transition probabilities between them. To control the MTF size, a fuzzy kernel average is used for each non-overlapping m*m window, that is, the transition probabilities of each sequence of length m are aggregated.

[0048]

[0049] Considering that sequence length affects computational efficiency, a piecewise aggregation approximation is introduced, employing an arithmetic progression method for parameter selection: the initial dimension is set to 8000, the fixed interpolation value is 400, and the value is progressively decreased to 400, resulting in 20 sets of parameters, namely [8000, 7600, ..., 400]. This reduces the sequence size while preserving the sequence trend. Therefore, for each set of FHR data, 20 MTF-based 2D images are ultimately obtained. The image size is set to 224*224, and the image modalities Mi∈Rn. i *d i This can be represented as M. i ∈R 20*224 Taking normal sample 1001 as an example, Figure 2 Display the corresponding MTF chart.

[0050] Text data extraction: The text data consists of clinical physiological information and morphological, temporal, and frequency domain parameters calculated from the FHR signal. The clinical physiological information selected includes two key pieces of information: maternal age and gestational age. The clinical morphological features of the FHR signal can be initially visually interpreted through the CTG paper report, which is currently the main diagnostic standard for obstetricians and gynecologists when interpreting CTG. This invention calculates parameters based on FIGO guidelines: the temporal domain features mainly originate from fetal heart rate variability (FHRV), i.e., variations in the heartbeat cycle. To analyze HRV, FHR needs to be converted to RR (heartbeat-by-heartbeat), and the conversion equation is shown in equation (3) below. Various statistical indicators are calculated by referring to the commonly used parameters of adult HRV to analyze the time domain characteristics of fetal heart rate variability signal. At the same time, the frequency range is divided into four frequency bands: very low frequency (VLF, 0~0.03Hz), low frequency (LF, 0.03~0.15Hz), mid frequency (MF, 0.15~0.5Hz) and high frequency (HF, 0.5~1Hz), and the eight frequency domain characteristics of FHRV are calculated. The specific indicators are calculated as shown in Table 1 below.

[0051]

[0052] Table 1. Definitions of morphological, temporal, and frequency domain characteristics of FHR signals

[0053]

[0054] (2) Image-Text Feature Fusion Network (ITFN)

[0055] ViT-based image encoder: The image encoder structure used in this invention is as follows Figure 3As shown, the process includes three stages: The first stage segments the FHR image after MTF conversion into 16×16 image blocks, which are then ordered sequentially to form a linear embedding sequence. This sequence is input into the Transformer Encoder to implement the standard Transformer application, while positional embeddings are added to remember the positional relationships between these blocks. The second stage uses the Transformer Encoder as the basic network module, mainly including Layer Norm, multi-head attention, droppath, and MLP Block structure. The multi-head attention mechanism calculates at each position based on the entire input FHR image sequence information, acquiring global feature information in shallow layers, learning high-quality intermediate features in intermediate layers, and retaining more comprehensive spatial information in deep layers, thus capturing more global features. By repeatedly stacking the Transformer Encoder 12 times, the network structure is continuously deepened to mine typical features of 2D image modal data. The final stage uses the standard MLP head mechanism to output the encoded single-modality image feature vector M. i .

[0056] CNN-based Text Encoder: This invention designs a 1D convolutional neural network as a text encoder for feature extraction from text data. The encoder mainly consists of convolutional layers, pooling layers, and fully connected layers, with its structure and parameter settings as follows: Figure 4 As shown, firstly, two convolutional layers are used for feature mapping, and a max-pooling layer is set to achieve modal feature extraction and dimensionality compression. Secondly, the multidimensional output data is flattened into a one-dimensional fully connected layer using a flattening method. Finally, two linear layers are used to map the flattened feature vectors to a higher-dimensional space and perform nonlinear transformations. To increase the model's nonlinearity and generalization ability, a ReLU activation function is introduced between each fully connected layer. Finally, the feature vector processed by the fully connected layers is used as the final output, i.e., the text modal feature vector M. t .

[0057] Weighted fusion: Based on the original early fusion, this invention adds random weights g to the multimodal feature vectors. This method improves upon simple concatenation by "dynamically" interacting with different modal data, allowing the model to fully learn the connections between different feature cascades. The specific expressions are shown in (4) to (5). The resulting multimodal feature vector Z is obtained after full interaction. f Then, a fully connected layer is used again for dimensionality reduction, finally achieving the classification of normal and pathological samples.

[0058] Z f =gM i +(1-g)M t (4)

[0059]

[0060] Where W1 and W2 are the learnable weight matrices of two different linear layers; σ(·) represents the activation function, which in this embodiment uses tanh (hyperbolic tangent function), Z f This represents the feature vector containing information from both modalities after fusion. To verify the superiority of ITFN, we compared different image backbone models, specifically AlexNet, VGG, and ResNet50. During the comparison, we only replaced the image encoder while keeping the text module and fusion method unchanged. We trained and tested on the same dataset using the same hyperparameter settings and training strategies, such as learning rate, batch size, and optimizer, to ensure the fairness and reliability of the experiment. As shown in Table 2, ITFN has a significant advantage, with an average accuracy of 0.96 on the test machine, which is 4% or more higher than other models.

[0061] Table 2 Comparison of test accuracy of different models

[0062] Model Test set accuracy ITFN (the present invention) 96.02％ ResNet50 92.31％ VGG 93.84％ AlexNet 74.24％

[0063] Table 3 summarizes various diagnostic methods proposed by researchers in the field of fetal monitoring over the past few years, covering both unimodal and multimodal studies. It compares the feasibility of multimodal features and highlights the superiority of the multimodal method of this invention. To ensure a fairer comparison, results from the same dataset were collected. As multimodal methods for extracting different modal features, existing technologies 1 and 2 are similar to this invention. For example, the former converts one-dimensional signals into two-dimensional images but does not fuse other modal information, resulting in a 16% difference in results. The latter only calculates the time-domain, frequency-domain, and time-frequency-domain features of the FHR signal and uses a machine learning algorithm, achieving an accuracy of 94.75%. Similarly, as multimodal methods, existing technologies 3 and 4 use different modalities and fusion methods compared to this invention, resulting in a 10% difference in results. Existing technology 5 extracts features from both FHR and UC signals simultaneously, without considering clinical information, resulting in a 1.07% accuracy gap compared to this invention. Existing technology 6 trains the model using only text modal data and employs a machine learning method, achieving approximately 3% lower accuracy than this invention. In summary, the ITFN network of this invention demonstrates better performance after the fusion of multimodal data and multiple models.

[0064] Table 3. Differences in diagnostic results among different network models

[0065]

[0066] The sources of the existing technologies are as follows:

[0067] Existing technology 1: Predicting fetal hypoxia using common spatial pattern and machine learning from cardiotocography signals

[0068] Existing technology 2: An attention-based CNN-BiLSTM hybrid neural network enhanced with features ofdiscrete wavelet transformation for fetal acidosis classification

[0069] Prior Art 3: Intelligent classification of antenatal cardiotocographysignals via multimodal bidirectional gated recurrent units

[0070] Existing technology 4: Intelligent antepartum fetal monitoring via deep learning and fusion of cardiotocographic signals and clinical data

[0071] Existing technology 5: A CNN-RNN unified framework for intrapartum cardiotocograph classification

[0072] Prior Art 6: Cardiotocography Data Analysis for Fetal HealthClassification Using Machine Learning Models

[0073] The above embodiments are not intended to limit the present invention, and the present invention is not limited to the above embodiments. Any embodiment that meets the requirements of the present invention is within the protection scope of the present invention.

Claims

1. A fetal assisted diagnostic system based on multimodal medical information fusion, characterized in that, include: The signal preprocessing module removes noise from the original fetal heart rate (FHR) signal to obtain a clean FHR signal. The image generation module converts the clean fetal heart rate (FHR) signal into image data; the image generation module uses Markov transfer field (MTF). The text extraction module extracts the parameter features of the clean fetal heart rate (FHR) signal and combines them with clinical physiological information to obtain text data. The parameter features include morphological, time-domain, and frequency-domain parameters, and the clinical physiological information includes maternal age and gestational age. The multimodal fusion module utilizes the Image-Text Feature Fusion Network (ITFN) to extract features from image and text data, obtaining the image feature vector M. i and text feature vector M t ; to transform the image feature vector M i and text feature vector M t Weighted fusion is performed to obtain the multimodal feature vector Z f The image-text feature fusion network (ITFN) includes an image encoder and a text encoder. The image encoder uses a ViT network, and the text encoder uses a convolutional neural network (CNN). The weighted fusion specifically involves: ； Where g represents the weights, W1 and W2 are learnable weight matrices for two different linear layers; σ(·) represents the activation function; Z f Represents a multimodal feature vector; The output module uses fully connected layers to process the multimodal feature vector Z. f Dimensionality reduction, outputting the classification results of normal and pathological samples.

2. The fetal assisted diagnostic system based on multimodal medical information fusion according to claim 1, characterized in that, The ViT network comprises the following sequentially connected components: The Patch Embedding module divides the input image into 16x16 patches, flattens each patch, and converts it into a one-dimensional vector representation. Position Embedding module: Adds position encoding to each patch to preserve the spatial location information of the image; The encoder group comprises 12 encoder blocks, each containing a multi-head self-attention mechanism module and a multilayer perceptron module. Each encoder block includes a first-level Layer Norm layer, a multi-head attention layer, and a first-level Dropout layer connected in sequence. The input of the encoder block is added to the output of the first-level Dropout layer and then fed into the second-level Layer Norm layer. The second-level Layer Norm layer is sequentially connected to the MLP linear layer and the second-level Dropout layer. The input of the second-level Layer Norm layer is added to the output of the second-level Dropout layer and then fed into the output of each encoder block. The multilayer perceptron module consists of two fully connected layers, with a GELU activation function used for nonlinear transformation in between, and includes a DropPath layer to prevent overfitting. Classification Head: The classification head includes a Pre-Logits layer and a linear layer; the output of the encoder group is fed into the classification head after passing through a Layer Norm layer, and the image feature vector M is output through the linear layer. i .

3. The fetal assisted diagnostic system based on multimodal medical information fusion according to claim 1, characterized in that, The convolutional neural network (CNN) comprises a first convolutional layer, a second convolutional layer, a max-pooling layer, and a fully connected layer connected in sequence. The fully connected layer outputs a text feature vector M. t The output of the max pooling layer is flattened by Flatten and then input into the fully connected layer.

4. An electronic device comprising a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the operation of the system according to any one of claims 1-3.

5. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the operation of the system according to any one of claims 1-3.

Images