Fetus auxiliary diagnosis system with multi-modal medical information fusion

Abstract

The invention discloses a fetal auxiliary diagnosis system based on multi-modal medical information fusion. The fetal auxiliary diagnosis system comprises a signal preprocessing module, an image generation module, a text extraction module, a multi-modal fusion module and an output module, wherein the multi-modal fusion module performs feature extraction on image data acquired by the image generation module and text data acquired by the text extraction module by using an ITFN (Inter-Text Feature Network) to obtain an image feature vector Mi and a text feature vector Mt; performing weighted fusion on the image feature vector Mi and the text feature vector Mt to obtain a multi-modal feature vector Zf; and the output module uses a full connection layer to carry out dimensionality reduction on the multi-modal feature vector Zf, and outputs a normal and pathological sample classification result. According to the method, different models are adopted to learn local semantics, more related knowledge information is obtained, effective fetal distress pathological diagnosis is achieved, the misdiagnosis rate is reduced, and the method has wide practical application value.

Description

Technical Field

[0001] The present invention relates to the field of medical health, in particular to the monitoring technology of pregnant women and fetuses, and specifically to a fetus auxiliary diagnosis system integrating multi-modal medical information. Background Art

[0002] With economic development, the number of older pregnant women has increased dramatically. Fetal acidosis caused by hypoxia can cause irreversible consequences such as organ damage or even death. Therefore, it is crucial to ensure the health of pregnant women and fetuses. This is a key research topic and breakthrough direction in the international medical and health field today and is of great significance.

[0003] In the late pregnancy, the fetus may suffer from hypoxia in the uterus, leading to acidosis, which may cause irreversible consequences such as fetal organ damage or even death. Therefore, it is crucial to ensure the health of pregnant women and fetuses. At present, cardiotocography (CTG) technology is a common method for obstetricians to assess whether the fetus is in distress and whether emergency intervention is needed. CTG technology assesses the status of the fetus by continuously and synchronously recording fetal heart rate (FHR) and uterine contraction (UC) signals. However, due to the dynamics and complexity of CTG signals, there are large differences in the interpretation of signals by different doctors, which may lead to unnecessary cesarean sections. In order to reduce the misdiagnosis rate, researchers have studied the intelligent cardiotocography (ICTG) method based on machine learning and deep learning tools, combined with guidelines such as the International Federation of Gynecology and Obstetrics, to intelligently assist in the diagnosis of fetal distress pathology. With the development of computer technology, ICTG methods are constantly being updated, from one-dimensional (1D) signal processing to feature extraction based on two-dimensional (2D) images, from traditional machine learning methods to deep learning methods, and from single-modal feature extraction to multi-modal feature considerations. However, the current technical challenges are mainly reflected in the following aspects:

[0004] (1) ICTG based on 1D signals mainly relies on mathematical analysis and pattern recognition of FHR signals. Although it has achieved a certain degree of intelligence, it has limitations in capturing the complex nonlinear relationships and potential patterns of sequence signals.

[0005] (2) Based on the combination of 2D images and deep learning, feature representation is learned from raw data, achieving better performance than traditional machine learning algorithms. However, ICTG based only on FHR is limited to specific information sources and cannot fully reflect the physiological condition of the fetus; and in clinical practice, obstetricians will comprehensively analyze the fetal status based on realistic factors such as maternal age and gestational age.

[0006] Therefore, a new auxiliary diagnosis system is urgently needed. Summary of the invention

[0007] The first purpose of the present invention is to solve the above-mentioned technical difficulties existing in the practical application of fetal assisted diagnosis technology, and to provide a fetal assisted diagnosis system with multimodal medical information fusion, which not only takes into account different data modalities, but also extracts features according to different model characteristics to achieve effective fetal distress pathology diagnosis and assist clinicians in monitoring distressed fetuses.

[0008] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0009] A fetal auxiliary diagnosis system based on multimodal medical information fusion, comprising:

[0010] The signal preprocessing module removes the noise of the original FHR signal to obtain a clean FHR signal;

[0011] Image generation module, converting clean FHR signals into image data;

[0012] The text extraction module extracts the parameter features of the clean FHR signal and combines it with clinical physiological information to obtain text data;

[0013] The multimodal fusion module uses the image-text feature fusion network ITFN to extract features from image data and text data to obtain the image feature vector M i and text feature vector M t ; The image feature vector M i and text feature vector M t Perform weighted fusion to obtain the multimodal feature vector Z f ; The image-text feature fusion network ITFN includes an image encoder and a text encoder, the image encoder adopts a ViT network, and the text encoder adopts a convolutional neural network CNN;

[0014] Output module, using a fully connected layer to transform the multimodal feature vector Z f Dimensionality reduction, output the results of normal and pathological sample classification.

[0015] Furthermore, the image generation module adopts Markov Transition Field (MTF).

[0016] Furthermore, the parameter characteristics include morphological, time domain, and frequency domain parameters, and the clinical physiological information includes maternal age and gestational age.

[0017] Furthermore, the ViT network includes:

[0018] Patch Embedding module: divides the input image into 16x16 patches, flattens each patch, and converts it into a one-dimensional vector representation;

[0019] Position Embedding module: adds position encoding to each patch to preserve the spatial position information of the image and ensure that the Transformer can exploit the position relationship;

[0020] Encoder group: including 12 layers of encoder blocks (Encoder Block), each encoder block includes a multi-head self-attention mechanism (Multi-HeadAttention) module and a multi-layer perceptron module (MLP Block), each layer of the encoder block includes a first-level LayerNorm layer, a multi-head attention layer and a first-level Dropout layer connected in sequence; the encoder block input is added to the output of the first-level Dropout layer and then input to the second-level Layer Norm layer; the second-level Layer Norm layer is sequentially connected to the MLP linear layer and the second-level Dropout layer; the input of the second-level Layer Norm layer is added to the output of the second-level Dropout layer as the output of each layer of the encoder block; the multi-layer perceptron module (MLP Block) consists of two fully connected layers, the GELU activation function is used in the middle for nonlinear transformation, and path discarding (DropPath) is included to prevent overfitting;

[0021] Classification head (MLP Head): The classification head includes a Pre-Logits layer and a linear layer; the output of the encoder group is input into the classification head after passing through a layer of Layer Norm, and the image feature vector M is output through the linear layer. i .

[0022] Furthermore, the convolutional neural network CNN includes a first convolutional layer, a second convolutional layer, a maximum pooling layer and a fully connected layer connected in sequence, and the fully connected layer outputs a text feature vector M t ; The output of the maximum pooling layer is flattened and input into the fully connected layer.

[0023] Furthermore, the weighted fusion is specifically:

[0024] Zf =gM i +(1-g)M t (4)

[0025]

[0026] Where g is the weight, W1 and W2 are the learnable weight matrices of two different linear layers; σ(·) represents the activation function; Z f Represents a multimodal feature vector.

[0027] A second object of the present invention is to provide an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above system.

[0028] The third object of the present invention is to provide a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned system.

[0029] The specific modules of the present invention are as follows:

[0030] First, in order to solve the problem of insufficient fetal pathology data, the present invention proposes a method for generating images using existing data sets to make up for the problem of insufficient data, thereby improving the generalization ability and robustness of the model.

[0031] Secondly, the present invention proposes a fetal diagnosis model (Multi-FHRNet) with multimodal data fusion. Taking into account the actual clinical factors, the model performance is further improved by integrating maternal age, gestational age and three types of features (morphology, time domain and frequency domain) of FHR signals as text data. By effectively utilizing various modal information, the experimental results show that the method of the present invention has achieved significant performance in determining whether the fetus is hypoxic.

[0032] In the third aspect, the present invention designs an Image-Text Fusion Net (ITFN), in which Vision Transformer (ViT) and Convolution Neural Networks (CNN) are used as image encoders and text encoders to extract different modal features. Furthermore, the adaptive weighting (ADW) of different modal feature vectors is implemented instead of the direct splicing method to effectively utilize various modal information.

[0033] Compared with the prior art, the beneficial effects of the present invention are at least:

[0034] The multimodal medical information fusion method provided in the embodiment of the present invention aims to improve the accuracy and efficiency of fetal pathology diagnosis through innovative technologies:

[0035] 1. The present invention constructs multimodal data based on image data and clinical information, integrates clinical related factors, and makes the diagnosis results more in line with actual medical needs. This construction of multimodal data not only improves the accuracy of diagnosis, but also enhances the generalization ability of the model, enabling it to be better applied in different clinical scenarios.

[0036] 2. In the present invention, ViT and convolutional neural network (CNN) process data of different modalities respectively, so that both local features and global features of data of different modalities can be processed more efficiently. ViT's attention mechanism can better capture the relationship between images, and CNN is good at capturing local features. The combination of the two provides a more comprehensive analysis for multimodal data. At the same time, an adaptive weighting method is used instead of direct feature vector splicing. By dynamically selecting and integrating feature information from different sources, the model performance is significantly improved and the utilization of multimodal information is optimized. Through the above method, the present invention not only improves the accuracy of fetal pathology diagnosis, but also enhances the generalization ability and clinical application value of the model. These innovations demonstrate the advancement and practicality of the present invention in the field of multimodal medical information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. The drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work.

[0038] Figure 1 The multimodal information fusion fetal auxiliary diagnosis architecture provided by the present invention;

[0039] Figure 2 The effect of converting 1D signal to 2D image used in the present invention;

[0040] Figure 3 The ViT-based picture data encoder provided by the present invention;

[0041] Figure 4 The text modality encoder network structure based on CNN provided by the present invention; DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The fetal assisted diagnosis system based on multimodal medical information fusion has great practical application value and can provide guidance for the existing medical and health fields to monitor whether the fetus is in distress, hypoxia, and whether human intervention is needed.

[0044] Figure 1 The multimodal medical information fusion architecture provided by the present invention includes:

[0045] (1) Data preprocessing

[0046] Noise reduction: FHR signals are usually obtained clinically by Doppler ultrasound probes or fetal scalp electrodes, and are easily affected by noise during the acquisition process. Fetal or maternal movement, sensor failure, and other phenomena can cause signal loss. The noise of FHR signals usually manifests as artifacts (or spikes) and missing (periods when the FHR value is zero), so the following three steps are used to reduce these two types of noise: First, when the FHR signal value is 0 and the duration is greater than 15s, it is directly removed, otherwise it is linearly interpolated; secondly, when the FHR signal is unstable, that is, the absolute value of two adjacent points is greater than 25bpm (|x(i)-x(i-1)|≥25), interpolation is performed between the starting sampling point and the first point of the next stable part; finally, when the FHR value is greater than 200bpm (x(i)≥200) or less than 50bpm (x(i)≤50), cubic spline interpolation is used for filling.

[0047] Convert to image: The present invention realizes the conversion of 1D signal data to 2D image data through MTF. First, the FHR is divided into Q quantile intervals according to its value range; secondly, a Q×Q Markov transfer matrix W is established, as shown in formula (1), w ij represents the transition frequency from bin i to bin j. Finally, the Markov transition field M is constructed as shown in formula (2), M ij Represented as any two time point data q i ,q j To control the MTF size, fuzzy kernel averaging is used for each non-overlapping m*m window, that is, the transition probability of each sequence of length m is aggregated.

[0048]

[0049] Considering that the sequence length will affect the computational efficiency, the segmented aggregation approximation is introduced, and the arithmetic method is used to obtain the parameters: the initial dimension is set to 8000, the fixed interpolation value is 400, and then it is reduced to 400 in sequence, and a total of 20 sets of parameters are obtained, namely [8000, 7600, ... 400], thereby reducing the sequence size while retaining the sequence trend. Therefore, for each set of FHR data, 20 2D images based on MTF can be obtained in the end. Set the image size to 224*224, and the image modality Mi∈R n i *d i , which can be expressed as M i ∈R 20*224 Taking normal sample 1001 as an example, Figure 2 The corresponding MTF graph is shown.

[0050] Text data extraction: Text data contains clinical physiological information as well as morphological, time domain, and frequency domain parameters calculated by FHR. The clinical physiological information selects two key information: maternal age and gestational age. The clinical morphological characteristics of the FHR signal can be preliminarily visually interpreted through the CTG paper report, which is the main diagnostic criterion for obstetricians and gynecologists when interpreting CTG. The present invention calculates parameters based on the FIGO guidelines: the time domain characteristics are mainly derived from fetal heart rate variability (FHRV), that is, heart rate cycle variability. To analyze HRV, FHR needs to be converted into RR (heartbeat-by-heartbeat). The conversion equation is shown in equation (3). Various statistical indicators are calculated by referring to the commonly used parameters of adult HRV to analyze the time domain characteristics of fetal heart rate variability signals. At the same time, the frequency range is divided into four frequency bands: very low frequency (VLF, 0-0.03 Hz), low frequency (LF, 0.03-0.15 Hz), medium frequency (MF, 0.15-0.5 Hz) and high frequency (HF, 0.5-1 Hz). The eight frequency domain characteristics of FHRV are calculated. The specific indicator calculation is shown in Table 1.

[0051]

[0052] Table 1 Definitions of FHR signal morphological features, time domain features, and frequency domain features

[0053]

[0054] (2) Image-Text Feature Fusion Network (ITFN)

[0055] Image encoder based on ViT: The image encoder structure used in the present invention is as follows Figure 3As shown in the figure, it includes three stages: in the first stage, the FHR image after MTF conversion is divided into image blocks of size 16×16 and sorted in sequence to form a linear embedding sequence, which is input into the Transformer Encoder to realize the application of the standard Transformer, and position embedding is added to memorize the position relationship between these blocks; in the second stage, the Transformer Encoder is used as the basic network module, which mainly includes Layer Norm, multi-head attention, droppath and MLP Block structure, in which the multi-head attention mechanism is calculated based on the entire input FHR image sequence information at each position, obtains global feature information at the shallow layer, learns high-quality intermediate features in the middle layer, and retains more comprehensive spatial information in the deep layer, thereby capturing more global features. By repeatedly stacking the Transformer Encoder 12 times to continuously deepen the network structure, the typical features of 2D image modal data are mined; in the last stage, the standard MLP head mechanism is used to output the encoded single-modal image feature vector M i .

[0056] CNN-based text encoder: This paper designs a 1D convolutional neural network as a text encoder for feature extraction of text data. The encoder is mainly composed of a convolutional layer, a pooling layer, and a fully connected layer. Its structure and parameter settings are as follows: Figure 4 As shown in the figure. First, two convolutional layers are used for feature mapping, and a maximum pooling layer is set to realize modal feature extraction and dimensionality compression; secondly, the output multidimensional data is flattened into one dimension and fully connected through the flattening method; finally, two linear layers are used to map the flattened feature vector to a higher dimensional space and perform nonlinear transformation. In order to increase the nonlinear ability and generalization ability of the model, the ReLU activation function is introduced between each fully connected layer. Finally, the feature vector processed by the fully connected layer is used as the final output, that is, the text modal feature vector M t .

[0057] Weighted fusion: Based on the original early fusion, the present invention adds random weights g to the multimodal feature vector. This method improves simple splicing and makes the interaction of different modal data "dynamic", so that the model can fully learn the connection between different feature cascades. The specific expressions are shown in (4) to (5). The multimodal feature vector Z after full interaction is obtained f Finally, the fully connected layer is used again to reduce the dimension, and finally the results of normal and pathological sample classification are achieved.

[0058] Z f =gM i +(1-g)M t (4)

[0059]

[0060] Where W1 and W2 are two different linear layer learnable weight matrices; σ(·) represents the activation function, and tanh (hyperbolic tangent function) is used in this embodiment, and Z f Represents the feature vector containing two modal information after fusion. In order to verify the superiority of ITFN, we compared different image backbone models, including AlexNet, VGG, and ResNet50. In the comparison process, we only replaced the image encoder part to ensure that the text module and fusion method remained unchanged, and trained and tested on the same dataset, using the same hyperparameter settings and training strategies, such as learning rate, batch size, optimizer, etc., to ensure the fairness and reliability of the experiment. As can be seen from Table 2 below, ITFN has a clear advantage, and its average accuracy on the test machine reached 0.96, which is 4% or more higher than other models.

[0061] Table 2 Comparison of test accuracy of different models

[0062] Model Test set accuracy ITFN (Invention) 96.02％ ResNet50 92.31％ VGG 93.84％ AlexNet 74.24％

[0063] Table 3 below summarizes the various diagnostic methods proposed by researchers in the field of fetal monitoring in the past few years, including single-modal and multi-modal studies, which are used to compare the feasibility of multi-modal features and highlight the superiority of the multi-modal method of the present invention. In order to make the comparison fairer, the invention results under the same data set are collected. As a multi-modal method to extract different modal features, the prior art 1 and 2 are similar to the present invention. For example, the former also converts the one-dimensional signal into a two-dimensional image but does not fuse other modal information, resulting in a 16% difference in results, while the latter only calculates the time domain, frequency domain and time-frequency domain features of the FHR signal and uses a machine learning algorithm with an accuracy of 94.75%. Also as a multi-modal method, the prior art 3 and 4 use different modalities and fusion methods with the present invention, and the results differ by 10%; the prior art 5 extracts the features of the FHR signal and the UC signal at the same time, without considering clinical information, and has an accuracy gap of 1.07% with the results of the present invention. The prior art 6 is a machine learning method that uses only text modal data to train the model, which is about 3% lower than the results of the present invention. In summary, the ITFN network of the present invention embodies better effects after the fusion of multimodal data and multi-models.

[0064] Table 3 Differences in network diagnosis results of different models

[0065]

[0066] The sources of the prior art are as follows:

[0067] Existing technology 1: Predicting fetal hypoxia using common spatial pattern and machine learning from cardiotocography signals

[0068] Existing technology 2: An attention-based CNN-BiLSTM hybrid neural network enhanced with features ofdiscrete wavelet transformation for fetal acidosis classification

[0069] Prior Art 3: Intelligent classification of antenatal cardiotocographysignals via multimodal bidirectional gated recurrent units

[0070] Existing technology 4: Intelligent antepartum fetal monitoring via deep learning and fusion of cardiotocographic signals and clinical data

[0071] Prior Art 5: A CNN-RNN unified framework for intrapartum cardiotocograph classification

[0072] Prior Art 6: Cardiotocography Data Analysis for Fetal HealthClassification Using Machine Learning Models

[0073] The above embodiments are not limitations of the present invention, and the present invention is not limited to the above embodiments. As long as the requirements of the present invention are met, they belong to the protection scope of the present invention.

Claims

1. A fetal auxiliary diagnosis system based on multimodal medical information fusion, characterized in that: include: The signal preprocessing module removes the noise of the original FHR signal to obtain a clean FHR signal; Image generation module, converting clean FHR signals into image data; The text extraction module extracts the parameter features of the clean FHR signal and combines it with clinical physiological information to obtain text data; The multimodal fusion module uses the image-text feature fusion network ITFN to extract features from image data and text data to obtain the image feature vector M i and text feature vector M t ; The image feature vector M i and text feature vector M t Perform weighted fusion to obtain the multimodal feature vector Z f ; The image-text feature fusion network ITFN includes an image encoder and a text encoder, the image encoder adopts a ViT network, and the text encoder adopts a convolutional neural network CNN; Output module, using a fully connected layer to transform the multimodal feature vector Z f Dimensionality reduction, output the results of normal and pathological sample classification.

2. The fetal auxiliary diagnosis system based on multimodal medical information fusion according to claim 1, characterized in that: The image generation module adopts the Markov transition field MTF.

3. The fetal auxiliary diagnosis system based on multimodal medical information fusion according to claim 1, characterized in that: The parameter characteristics include morphological, time domain, and frequency domain parameters, and the clinical physiological information includes maternal age and gestational age.

4. The fetal auxiliary diagnosis system based on multimodal medical information fusion according to claim 1, characterized in that: The ViT network includes: Patch Embedding module: divides the input image into 16x16 patches, flattens each patch, and converts it into a one-dimensional vector representation; Position Embedding module: adds position encoding to each patch to preserve the spatial position information of the image; Encoder group: including 12 layers of encoder blocks, each encoder block includes a multi-head self-attention mechanism module and a multi-layer perceptron module, each layer of encoder block includes a first-level Layer Norm layer, a multi-head attention layer and a first-level Dropout layer connected in sequence; the encoder block input is added to the output of the first-level Dropout layer and then input to the second-level Layer Norm layer; the second-level Layer Norm layer is sequentially connected to the MLP linear layer and the second-level Dropout layer; the input of the second-level Layer Norm layer and the output of the second-level Dropout layer are added as the output of each layer of encoder block; the multi-layer perceptron module consists of two fully connected layers, the GELU activation function is used in the middle for nonlinear transformation, and the DropPath layer is included to prevent overfitting; Classification head: The classification head includes a Pre-Logits layer and a linear layer; the output of the encoder group is input into the classification head after passing through a layer of LayerNorm, and the image feature vector M is output through the linear layer. i .

5. The fetal auxiliary diagnosis system based on multimodal medical information fusion according to claim 1, characterized in that: The convolutional neural network CNN includes a first convolutional layer, a second convolutional layer, a maximum pooling layer and a fully connected layer connected in sequence, and the fully connected layer outputs a text feature vector M t ; The output of the maximum pooling layer is flattened and input into the fully connected layer.

6. The fetal auxiliary diagnosis system based on multimodal medical information fusion according to claim 1, characterized in that: The weighted fusion is specifically as follows: Z f =gM i +(1-g)M t (4) Where g is the weight, W1 and W2 are the learnable weight matrices of two different linear layers; σ(·) represents the activation function; Z f Represents a multimodal feature vector.

7. An electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the system according to any one of claims 1 to 6.

8. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the system described in any one of claims 1-6.

Images