Fetal heart rate monitoring analysis method and monitoring equipment

Abstract

The invention relates to the technical field of fetal heart monitoring, in particular to a fetal heart monitoring analysis method, monitoring equipment and a storage medium. The fetal heart monitoring analysis method comprises the following steps: acquiring a heart rate signal and clinical table data of a fetus; performing continuous wavelet transform on the heart rate signal to obtain a time-frequency domain image, and inputting the time-frequency domain image into a ResNet50 network and a Mangban module to obtain image time sequence characteristics; inputting the clinical table data into a multi-scale text encoder, obtaining dependency relationships of the clinical table data in time sequence data of different scales, and fusing the dependency relationships to obtain time sequence dependency features; fusing the time sequence features and the time sequence dependence features of the image by adopting a cross-modal attention mechanism to obtain fused features; the fusion features are input into a classification module, an anoxia risk assessment result is obtained, and a joint loss function of the classification module comprises intra-modal alignment loss, inter-modal alignment loss and masking language modeling loss. According to the invention, the accuracy and reliability of diagnosis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of fetal heart monitoring, in particular to a fetal heart monitoring analysis method, monitoring equipment and storage medium. Background Art

[0002] The importance of cardiotocography (CTG) in obstetrics is self-evident. It evaluates the health of the fetus, especially the risk of fetal hypoxia, by monitoring the fetal heart rate (FHR) and uterine contraction (UC). The interpretation of CTG signals usually depends on the experience and subjective judgment of medical staff. The consistency of CTG signals may be poor, and the same CTG result may be interpreted as different conclusions by different doctors, which will affect the accuracy and reliability of the diagnosis. Summary of the invention

[0003] The technical problem to be solved by the embodiments of the present invention is to provide a fetal heart monitoring analysis method and a monitoring device to solve the problem of insufficient accuracy and reliability of diagnosis in the prior art.

[0004] The present invention discloses a fetal heart monitoring analysis method, which is applied to a fetal heart monitoring analysis model, wherein the fetal heart monitoring analysis model includes a continuous wavelet transform module, at least one group of interconnected ResNet50 networks and Mamba modules, a multi-scale text encoder, a cross-modal attention module, and a classification module;

[0005] The fetal heart monitoring analysis method comprises:

[0006] Obtain fetal heart rate signals and clinical table data;

[0007] Performing continuous wavelet transform on the heart rate signal to obtain a time-frequency domain image, and inputting the time-frequency domain image into a ResNet50 network and a Mamba module to obtain image temporal features;

[0008] Inputting the clinical table data into a multi-scale text encoder, obtaining dependency relationships in time series data of the clinical table data at different scales, and fusing the dependency relationships to obtain time series dependency features;

[0009] A cross-modal attention mechanism is used to fuse the image temporal features and the temporal dependency features to obtain fused features;

[0010] The fused features are input into a classification module to obtain a hypoxia risk assessment result, and a joint loss function of the classification module includes intra-modality alignment loss, inter-modality alignment loss and masked language modeling loss.

[0011] Optionally, the step of performing continuous wavelet transform on the heart rate signal to obtain a time-frequency domain image comprises:

[0012] Mapping the heart rate signal to a two-dimensional time-frequency domain space, adjusting the frequency and time resolution by scaling and translating factors, and obtaining a two-dimensional matrix;

[0013] The two-dimensional matrix is ​​normalized and pseudo-colored to be converted into the time-frequency domain image in RGB format.

[0014] Optionally, the step of inputting the time-frequency domain image into a ResNet50 network and a Mamba module to obtain image temporal features includes:

[0015] The time-frequency domain image is adjusted to a preset size and then input into a ResNet50 network to obtain pattern and sequence features in the time-frequency domain image, and output a feature vector with a length of 2048;

[0016] The feature vector is processed by combining the Mamba module with a multi-layer perceptron and a SiLU activation function, and the feature vector is converted and integrated into a target vector with a length of 512.

[0017] Optionally, the step of applying the multi-scale text encoder comprises:

[0018] converting the clinical table data into a clinical data matrix;

[0019] For each row of data in the clinical data matrix, one-dimensional convolution is used to extract local features to obtain local linear features and basic relationships between adjacent features;

[0020] Using dilated convolution for the clinical data matrix to obtain long-range dependencies of different scales;

[0021] Cascade the outputs of one-dimensional convolution and dilated convolution to obtain splicing features;

[0022] The concatenated features are processed through an activation function and a fully connected layer to obtain the temporal dependency features.

[0023] Optionally, the multi-scale text encoder includes a one-dimensional convolution module, a hole one-dimensional convolution module and a multi-layer perceptron connected in sequence;

[0024] The one-dimensional convolution module includes a one-dimensional convolution block, a batch normalization and ReLU activation layer, and an average pooling layer connected in sequence;

[0025] The atrous one-dimensional convolution module includes one-dimensional convolution blocks and batch normalization and ReLU activation layers connected in sequence;

[0026] The convolution kernel size of the one-dimensional convolution block in the one-dimensional convolution module is k=3, the step size is s=i, and the padding is p=1;

[0027] The convolution kernel size of the one-dimensional convolution block in the hole one-dimensional convolution module is k=3, the step size is s=i, the padding is p=1, and the expansion parameter is d=i;

[0028] Among them, i is the number of layers of the one-dimensional convolutional block.

[0029] Optionally, the step of fusing the image temporal features and the temporal dependent features using a cross-modal attention mechanism includes:

[0030] Embedding the image temporal features and the temporal dependency features into a unified feature space to form a unified feature;

[0031] The unified features are transformed linearly to generate a query matrix, a key matrix and a value matrix;

[0032] The correlation among the query matrix, key matrix and value matrix is ​​calculated. Based on the correlation, the query, key and value are divided into multiple different heads through a multi-head attention mechanism. Each head calculates attention independently and is fused by splicing.

[0033] Optionally, the step of inputting the fusion features into a classification module comprises:

[0034] Mapping the fusion features to the risk assessment space through a fully connected layer to obtain mapping features, inputting the mapping features into a softmax function to generate a probability distribution, and obtaining the probability of belonging to each health category;

[0035] Before the step of inputting the fusion feature into the classification module, the method further includes:

[0036] The classification module is optimized by defining a cross entropy loss function.

[0037] Optionally, the heart rate signal is recorded in the form of time series sequence data, including: time index, fetal heart rate at each quarter-second interval, uterine contraction intensity at each interval, seconds elapsed from the start of the monitoring period, and minutes elapsed from the start of the monitoring period;

[0038] The clinical table data includes: clinical data of the delivery process used for excluding data, clinical data of the delivery process not involved in the exclusion but used for recording, clinical data of the delivery results, and fetal heart rate monitoring signal standards;

[0039] The clinical data of the delivery process used to exclude data include: maternal age, gestational age, known fetal diseases, pregnancy type, and delivery type;

[0040] The clinical data of the delivery process that were not included in the exclusion but were used for recording included: fetal sex, number of deliveries, risk factors, complications during pregnancy, saline-related indicators during pregnancy, medications, whether the delivery was induced, type of fetal position, stagnation during delivery, head-body incoordination, and cranio-pelvic incoordination;

[0041] The clinical data of delivery outcomes described included: umbilical artery pH, partial carbon dioxide pressure, base excess, basal deficit in extracellular fluid, neonatal assessment, number of days in the neonatal intensive care unit, seizures, hypoxic-ischemic encephalopathy, intubation, and Apgar score;

[0042] The fetal heart rate monitoring signal standards include: duration of the first stage of labor, duration of the second stage of labor, type of measuring equipment, and time interval from end of signal to delivery.

[0043] Optionally, the fetal heart monitoring analysis method further comprises: training the fetal heart monitoring analysis model through training data;

[0044] The training data covers five dimensions: discharge records, medical records, ultrasound reports, delivery records, and neonatal records; the training data contains time series data, which records multiple key time nodes, including the time nodes of maternal admission, delivery, fetal birth, blood gas analysis, and fetal heart rate monitoring;

[0045] The discharge records include: the Apgar score of the newborn at 1 minute after birth, the Apgar score of the newborn at 5 minutes after birth, the mother's age, weeks of pregnancy, the total number of pregnancies of the female, the total number of births of the female, whether the baby is macrosomia, whether the fetal growth is restricted, the baby's birth weight, gender, whether the newborn is asphyxia, whether the baby is anemic, whether the baby has gestational hypertension, whether the baby has diabetes, whether the baby has hyperthyroidism, whether the baby has hypothyroidism, and whether the baby has premature rupture of membranes;

[0046] The medical records include: admission systolic blood pressure, admission diastolic blood pressure, presence or absence of fever, and whether induced labor was performed;

[0047] The ultrasound report includes: amniotic fluid dark area, amniotic fluid index, umbilical blood flow systolic / diastolic ratio;

[0048] The delivery record includes: the number of umbilical cord circles around the neck, the number of umbilical cord twists, whether there is a single umbilical artery, whether there is meconium in the amniotic fluid, fetal position, duration of the first stage of labor, stagnation during labor, duration of the second stage of labor, type of delivery, and type of intrapartum monitoring;

[0049] The neonatal records include: umbilical artery blood pH value, umbilical artery blood base excess, umbilical artery standard bicarbonate concentration, and partial pressure of carbon dioxide in umbilical artery blood.

[0050] The present invention also discloses a monitoring device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0051] Compared with the prior art, the beneficial effects of the fetal heart monitoring and analysis method provided by the embodiment of the present invention are: combining the fetal heart rate signal and clinical table data, comprehensively utilizing different types of information sources, and enhancing the understanding of the correlation between different information. The multimodal fusion method helps to consider various aspects of information more comprehensively, so that the model can more comprehensively and accurately evaluate the health status of the newborn, thereby improving the accuracy and reliability of the diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The scheme of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments, in which:

[0053] Figure 1 It is a flow chart of an embodiment of the fetal heart monitoring and analysis method provided by the present invention;

[0054] Figure 2 It is a structural schematic diagram of an embodiment of a fetal heart monitoring and analysis model provided by the present invention;

[0055] Figure 3 It is a structural diagram of the ResNet50 module provided by the present invention;

[0056] Figure 4 is a schematic diagram of the structure of the multi-scale text encoder provided by the present invention;

[0057] Figure 5 is a schematic diagram of the structure of the cross-modal attention module provided by the present invention;

[0058] Figure 6 A schematic diagram of an embodiment of fetal heart rate and uterine contraction signal data;

[0059] Figure 7 A schematic diagram of another embodiment of fetal heart rate and uterine contraction signal data. DETAILED DESCRIPTION

[0060] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. Now, in conjunction with the accompanying drawings, the preferred embodiments of the present invention are described in detail.

[0061] Please refer to Figure 1 and Figure 2 , Figure 1 is a flow chart of an embodiment of the fetal heart monitoring and analysis method provided by the present invention, Figure 2FIG. 1 is a schematic diagram of the structure of an embodiment of the fetal heart monitoring analysis model provided by the present invention. Figure 2 As shown, the fetal heart monitoring analysis model provided by the present invention includes a continuous wavelet transform module, and at least one group of interconnected ResNet50 networks and Mamba modules, a multi-scale text encoder, a cross-modal attention module and. Among them, the continuous wavelet transform module is used to receive the heart rate signal, generate the corresponding time-frequency domain image according to the heart rate signal, and input the time-frequency domain image into at least one group of interconnected ResNet50 networks and Mamba modules to obtain the image timing features. The multi-scale text encoder is used to receive clinical data tables and calculate the timing dependency features of the clinical data tables. The image timing features and the timing dependency features are input into the cross-modal attention module for fusion, and the fused features are output. The fused features are input into the classification module to obtain the hypoxia risk assessment results. Please refer to the following for the specific analysis process:

[0062] S101: Acquire fetal heart rate signals and clinical table data.

[0063] In a specific implementation scenario, the fetal heart rate signal is usually obtained through a fetal heart monitor. The fetal heart rate signal is recorded by the fetal heart monitor and is usually stored in a computer or data recording device in digital form. These data can be obtained by connecting to the monitor's interface or downloading the stored file. Clinical table data is obtained through medical records, and the clinical table data usually contains information about the health status of the pregnant woman and the fetus, such as basic information of the pregnant woman, pregnancy examination results, medication records, etc.

[0064] In one implementation scenario, CTG data is recorded in the form of time series data, including the parameters recorded in Figure 1:

[0065]

[0066] Table 1

[0067] Recording data as time series provides greater flexibility for subsequent data analysis, allowing for various time series analysis, model building and prediction, and convenient data visualization by drawing time series charts. By recording the number of seconds and minutes that have passed since the start of the monitoring period, data from different time points can be easily associated and compared. Recording fetal heart rate and contraction intensity at quarter-second intervals can provide very fine time resolution, which helps track instantaneous changes in fetal heart rate and contraction intensity and capture possible short-term changes.

[0068] Clinical data is recorded in the form of a structured table, and its characteristics can be further divided into four parts: clinical data characteristics of the delivery process used to exclude data (clinical criteiria), clinical data characteristics of the delivery process that are not involved in exclusion but used for recording (additional clinical criteria), clinical data characteristics of delivery outcomes (outcome criteria), and fetal heart rate monitoring (CTG) signal criteria (signal criteria). The four together constitute the complete data features in the clinical data set. Specifically, as shown in Table 2:

[0069]

[0070]

[0071]

[0072] Table 2

[0073] Recording different types of clinical data by category will help better integrate the data and facilitate subsequent analysis and research. Categorizing the clinical data of the delivery process used to exclude data and the clinical data of the delivery process that are not involved in the exclusion but are used for recording will make it easier to identify which data are used to exclude interference factors and which data are used to record important information when analyzing the data.

[0074] S102: Perform continuous wavelet transform on the heart rate signal to obtain a time-frequency domain image, and input the time-frequency domain image into the ResNet50 network and the Mamba module to obtain image temporal features.

[0075] In a specific implementation scenario, the heart rate signal is preprocessed, such as denoising, filtering and other operations, to ensure data quality. The preprocessed heart rate signal is subjected to continuous wavelet transform to obtain a time-frequency domain image. The continuous wavelet transform converts the heart rate signal into a time-frequency domain image, revealing the characteristic changes of the heart rate signal in time and frequency. The one-dimensional heart rate signal F(t) can be mapped to a two-dimensional time-frequency domain space by applying CWT (Continuous Wavelet Transform). The calculation formula is:

[0076]

[0077] Among them, ψ(t) is the mother wavelet function. A suitable mother wavelet function can be selected according to actual needs. Commonly used mother wavelet functions include Morlet wavelet, Mexican Hat wavelet, etc. The scaling factor a controls the frequency of the wavelet function, and the translation factor b controls the position of the wavelet function in time. By adjusting the values ​​of a and b, the signal can be analyzed at different scales in the time-frequency domain. The heart rate signal F(t) is inner-producted with the translated and scaled version of the mother wavelet function ψ(t), and the entire time range is integrated to obtain the wavelet coefficients W(a, b), which represent the time-frequency information of the signal at different scales a and different times b. In this embodiment, W(a, b) is a two-dimensional matrix, in which rows represent different scales and columns represent different times, and its typical size is 512*512.

[0078] Normalize and pseudo-color W(a, b) to better present the time-frequency information. Specifically, linearly normalize the values ​​in W(a, b) so that their value range is between [0, 1] or [0, 255]. Determine the color mapping relationship used to represent different values, usually using a heat map (transition from cold to warm colors) or a rainbow color scale. Based on the normalized values, map each value to a corresponding color to generate a color image, and convert the pseudo-color processed image to RGB format to generate a time-frequency domain image.

[0079] Input the time-frequency domain image into the ResNet50 network and the Mamba module to obtain the image time series features. ResNet50 is a deep convolutional neural network with a depth of 50 layers. Through multiple layers of convolution and pooling layers, ResNet50 can learn and extract abstract features in the image, thereby capturing the deep features in the time-frequency domain image. The Mamba module uses SSSM (Selective Structured State Space Models), which can better capture the complex state transitions and dynamic patterns in time series data, and help to more accurately capture the characteristics of subtle changes in the fetal heart rate signal. ResNet50 is responsible for extracting the deep features of the image, while the Mamba module further enhances the ability to parse the time series information through SSSM. The combined effect of these two models enables the system to comprehensively utilize image features and time series information to extract key features in the heart rate signal more comprehensively and accurately. Please refer to Figure 3 , Figure 3 It is a structural schematic diagram of the image feature extraction module provided by the present invention, and the image feature extraction module includes a continuous wavelet transform module and at least one group of interconnected ResNet50 networks and Mamba modules.

[0080] In one implementation scenario, the time-frequency domain image is preprocessed by standardization to ensure that the mean and variance of the image data are suitable for the input requirements of ResNet50. The processed time-frequency domain image is adjusted to the input size of the ResNet50 network, usually 224x224 pixels, and then input into the ResNet50 network.

[0081] In other implementation scenarios, the acquired preprocessed image is first input into the image block embedding layer, which is used to process local areas or blocks in the image data. In this embodiment, the image block embedding layer divides the entire input time-frequency domain image into small blocks, and performs feature extraction and encoding on each small block to obtain the corresponding image tensor so that the model can better understand the local structure and content of the image. Then input the Mamba module. The output of the Mamba module will be input into at least one set of interconnected ResNet50 and Mamba modules.

[0082] ResNet50 contains multiple deep convolutional layers. When processing time-frequency domain images, the deep convolutional layers can effectively capture the patterns and sequence features in the image. These features can include information such as signal patterns of different frequencies, temporal relationships, and time-frequency distributions, and extract feature vectors that are helpful for further processing. ResNet50 is a deep residual neural network structure that makes the network easier to train and improves the network's representation capabilities. It can better capture complex features in the data, thereby improving the efficiency and accuracy of feature extraction. For example, in this implementation scenario, each CWT image is extracted by ResNet50 to obtain a feature vector with a length of 2048 dimensions as a high-order semantic representation of the image.

[0083] The core of the Mamba module is based on the Selective State Space Model (SSSM), which captures the interaction between sequence features and timing information through state update equations and output equations. The state space equations are as follows:

[0084] State update equation:

[0085] x k+1 =A d x k +B d U k

[0086] Output equation:

[0087] y k =C d x k +D d U k

[0088] Among them, x k is the hidden state, U kis the input feature, y k is the output feature.

[0089] In the state update equation, by A d and B d The matrix describes how the hidden state is updated based on the previous state and the input features. This iterative update process of the state can pass on the timing information, allowing the model to capture the evolution between data points in the sequence. Through the state space model, the sequence features can be encoded into the hidden state. Hidden state x k Contains information about patterns, trends, or other important features in the sequence that can help capture the interaction between sequence features and time series information.

[0090] The output equation describes how to get from the hidden state x k And the input feature U k Generate observation data y k , this mapping relationship can help the model learn how to generate corresponding observations based on hidden states and input features, thereby capturing the association between sequence features and time series information.

[0091] Furthermore, the Mamba module is combined with a multi-layer perceptron (MLP) and a SiLU activation function to enhance the nonlinear expression capability. Each layer in the MLP usually contains a nonlinear activation function to enable the entire network to learn complex nonlinear relationships. SiLU is a nonlinear activation function that can capture more complex data patterns to a certain extent. Compared with simple linear activation functions (such as ReLU), SiLU can provide richer expression capabilities. The expression of SiLU is:

[0092]

[0093] Through the dynamic weight allocation mechanism, the Mamba module can flexibly adjust the contribution of features according to the characteristics of different tasks and input data, thereby improving the adaptability and generalization ability of the model. Through the dynamic weight allocation mechanism of the Mamba module, the feature vector extracted by ResNet50 is combined with dynamic modeling to generate a target vector that represents the comprehensive features of the time-frequency domain image. Feature fusion can better capture the association between sequence features and temporal information, improve the quality of feature representation and the performance of multimodal tasks. For example, in this implementation scenario, the length of the target vector finally generated for each CWT image is 512.

[0094] Combining CWT and ResNet50, richer feature information can be extracted in the frequency domain, and the deep structure of ResNet50 can be used to learn these features, thereby improving the feature extraction and representation learning capabilities of data such as heart rate signals. In frequency domain feature extraction, the Mamba module can help fuse the features extracted from CWT and ResNet50 to better capture the dynamic features and complex relationships in the data. Through dynamic feature fusion, the model can better understand the timing information and frequency domain characteristics in the data, thereby improving the analysis and recognition capabilities of data such as fetal heart rate signals. In summary, the combination of CWT and ResNet50 can provide higher temporal resolution and richer feature information in frequency domain feature extraction, while the Mamba module further optimizes the efficiency of dynamic feature fusion, enabling the model to better understand the dynamic features in complex data, and improving the performance and generalization of the model. Through the above process, the heart rate signal is characterized by high quality, providing powerful feature input for subsequent classification or multimodal fusion tasks, and improving the accuracy and stability of the prediction of neonatal hypoxia risk.

[0095] S103: Input the clinical table data into a multi-scale text encoder, obtain the dependency relationship of the clinical table data in the time series data at different scales, and fuse the dependency relationship to obtain the time series dependency feature.

[0096] In a specific implementation scenario, the clinical table data is preprocessed, including missing value processing, standardization, normalization and other operations to ensure data quality and consistency. The preprocessed clinical table data is used as the input MDTE (Multiscale Dynamic Time Evolution, multi-distance dependency encoding) module. The MDTE module combines the characteristics of one-dimensional convolution and dilated convolution to effectively extract multi-scale dependencies in clinical table data. Specifically, the MDTE module can use one-dimensional convolution layers to capture local features in the data, and can effectively learn feature patterns at different time steps in the data. The MDTE module expands the receptive field by increasing the interval of the convolution kernel through dilated convolution, thereby capturing dependencies of different scales in the data. Dilated convolutions with different dilation rates can effectively capture dependencies at different distances and improve the model's ability to learn long-distance dependencies in time series data. Combining one-dimensional convolution and dilated convolution, the MDTE module can extract dependencies in time series data at different scales. This can capture complex patterns and structures in the data more comprehensively.

[0097] In one implementation scenario, the clinical table data is converted into a clinical data matrix X∈R N×M, where n is the number of samples and M is the feature dimension of each sample. For each sample, extract features from the clinical table data. These features can be various clinical indicators, physiological parameters, disease diagnosis and other information, and each feature corresponds to a column in the table. For the categorical data corresponding to each feature, feature encoding is required to convert the categorical variable of the feature into a numerical variable, such as using one-hot encoding or label encoding. Arrange the feature values ​​of each sample in the feature order to form a feature vector. Arrange the feature vectors of all samples in rows to form a clinical data matrix X.

[0098] For each row of data x i ∈R M , the MDTE module performs preliminary local feature extraction through the 1D-CNN (one-dimensional convolution) layer, and its convolution operation is defined as:

[0099]

[0100] Where y[i] is the output feature value, w[k] is the convolution kernel weight, and K is the length of the convolution kernel. The convolution kernel slides along the feature dimension to capture local linear feature patterns. The initial processing of 1D-CNN can effectively identify the basic relationship between adjacent features in tabular data.

[0101] In order to further expand the receptive field of features, the MDTE module introduces dilated convolution (Dilated CNN). Dilated convolution introduces the dilation factor r in the convolution kernel and samples the input data in a jumpy manner to capture long-range dependencies of different scales. Its mathematical expression is:

[0102]

[0103] Where r is the expansion factor, which controls the receptive field of the convolution kernel. By adjusting r, the dilated convolution can capture long-range dependencies of different scales without increasing the computational complexity. The multi-scale nature of the dilated convolution is particularly important for modeling non-local relationships between features in tabular data.

[0104] In order to better combine multi-scale features, the MDTE module cascades and fuses the outputs of 1D-CNN and multi-layer dilated convolution. Assume that the convolution output of the lth layer is Among them C l Represents the number of channels of this layer, and the fused feature representation is obtained by stacking multiple layers of convolution:

[0105] H = Concat(H (1) , H (2) , …, H (L) )

[0106] Among them, Concat represents the feature concatenation operation, and L is the number of layers in the network. The concatenated feature H is further processed through the activation function (such as ReLU or SiLU) and the fully connected layer to generate the final high-dimensional feature representation, that is, the temporal dependency feature. Ultimately, the temporal dependency feature extracted by the MDTE module can effectively encode the key information in the clinical table data and provide rich representation for subsequent multimodal fusion.

[0107] Through the above steps, the MDTE module combines local feature extraction (1D-CNN) and multi-scale long-range dependency capture (Dilated CNN), which can effectively capture feature relationships of different scales in tabular data processing scenarios and improve the adaptability and generalization ability of the model.

[0108] Please refer to Figure 4 , Figure 4 It is a structural schematic diagram of the multi-scale text encoder provided by the present invention. The multi-scale text encoder includes a one-dimensional convolution module, a hole one-dimensional convolution module and a multi-layer perceptron (MLP) connected in sequence, and receives tabular data. The one-dimensional convolution block includes a three-layer structure: a one-dimensional convolution block, batch normalization and ReLU activation, and average pooling, which is used to extract local dependency features and capture local feature information. The convolution kernel size k=3, step size s=i, and padding p=1 of the one-dimensional convolution block in the one-dimensional convolution module. The hole one-dimensional convolution block adds an expansion parameter d (hole rate) in the convolution operation to expand the receptive field, thereby capturing long-distance relationships across features. The hole one-dimensional convolution module includes a one-dimensional convolution block and a batch normalization and ReLU activation layer connected in sequence; the convolution kernel size k=3, step size s=i, padding p=1, and expansion parameter d=i of the one-dimensional convolution block in the hole one-dimensional convolution module, wherein i is the number of layers of the one-dimensional convolution block. The outputs of the one-dimensional convolution module and the atrous one-dimensional convolution module are further transformed and concatenated through a multi-layer perceptron (MLP) to generate a unified feature representation, and finally output a high-dimensional feature vector for subsequent multimodal fusion or classification tasks.

[0109] Through such a structure, the clinical data feature extraction module can effectively extract local dependency features and long-distance dependency features from tabular data. By combining one-dimensional convolution blocks and void one-dimensional convolution blocks, combined with MLP for feature transformation and fusion, it generates high-dimensional feature representation and provides rich feature information for subsequent tasks.

[0110] Furthermore, the output of the multi-scale text encoder is a high-dimensional data. In order to remove unnecessary information in this high-dimensional data, dimensionality reduction can be performed through a module, which includes a fully connected layer, a Sigmoid activation function, a pruning layer, and another fully connected layer connected in sequence. The first fully connected layer linearly transforms the data encoded by the text encoder. The Sigmoid activation function maps the output value of the fully connected layer to between 0 and 1, introducing nonlinearity. The pruning layer deletes unimportant connections or parameters. The second fully connected layer further processes the data processed by the Sigmoid activation function to further extract features.

[0111] S104: Using a cross-modal attention mechanism to fuse the image temporal features and the temporal dependency features to obtain fused features.

[0112] In a specific implementation scenario, the image temporal features and the temporal dependency features are fused to obtain the fused features. The efficient interaction between the image temporal features and the temporal dependency features is achieved through multi-head attention (MHA). Specifically, the image temporal features are recorded as XFHR∈R n×dFHR , the temporal dependency feature is recorded as XClinical∈R m×dClinical , embed the image temporal features and temporal dependency features into a unified feature space to form a unified representation X∈R (n+m)×d , recorded as input features. The input features are transformed into query matrix (Q), key matrix (K) and value matrix (V) through linear transformation. The calculation formula is:

[0113] Q=XW Q , K = XW k , V = XW V

[0114] Among them, W Q , W k , W v is the learned projection matrix.

[0115] The correlation between the query matrix (Q), key matrix (K) and value matrix (V) is calculated through the attention mechanism. The specific formula is:

[0116]

[0117] Among them, d k Represents the feature dimension of a single attention head. T is the temperature parameter. The temperature parameter is used to adjust the "sharpness" of the attention distribution and affects the distribution of the attention weight.

[0118] The overall expression of multi-head attention is:

[0119] MultiHead(Q,K,V)=Concat(head1,...,head n )W O

[0120] Among them, the calculation of each attention head is head i =Attention(Q i , K i , V i ), is the output projection matrix. Through the multi-head mechanism, the model can capture the multimodal association between image temporal features and temporal dependency features from different subspaces. After multi-head attention processing, the fusion feature obtained is the matrix HFusion∈R (n+m)×d ,This matrix integrates the information of different modalities and is used in the subsequent classification module.

[0121] Please refer to Figure 5 , Figure 5 It is a structural schematic diagram of the cross-modal attention module provided by the present invention. The multi-scale text encoder performs a preliminary transformation on the features from two different modalities to generate query matrices Q1, Q2, key matrices K1, K2 and value matrices V1, V2. These matrices will be used for subsequent attention calculations. Calculate the similarity between the query of one modality and the key of another modality, use the attention mechanism to weight the corresponding values, and realize information interaction between different modalities, which can help the model focus on important features and reduce noise. The interactive features are calculated by multiple parallel attention heads, and each attention head can capture different feature relationships and enhance the expression ability of the model. The outputs of different attention heads are fused through the feature splicing layer to generate a unified feature representation, and the information of different heads is comprehensively utilized. The fused features are further processed by the fully connected layer to generate the final high-dimensional feature vector, including nonlinear transformation and adjustment of feature dimensions, to provide richer feature representations for subsequent tasks such as classification or regression.

[0122] This model architecture allows feature interaction and information fusion between different modalities, thereby improving the model's ability to model multimodal data. At the same time, it achieves adaptive attention to different features through the attention mechanism, which helps to improve the performance and generalization ability of the model.

[0123] S105: Input the fused features into the classification module to obtain the hypoxia risk assessment result. The joint loss function of the classification function includes intra-modality alignment loss, inter-modality alignment loss and masked language modeling loss.

[0124] In a specific implementation scenario, a classification module is constructed using a fully connected layer and a softmax activation function, and the fused features are input into the classification module to obtain the hypoxia risk assessment results. The fused feature representation is input into the fully connected layer for feature mapping and extraction. The fully connected layer can learn the nonlinear relationship between features, helping the model to better distinguish different categories. The Softmax function can convert the model output into the probability of each category, which facilitates the classification task. The Softmax activation function is used after the output of the fully connected layer to convert the output into a category probability distribution.

[0125] The classification module is optimized using a custom joint loss function, which includes intra-modality alignment loss, inter-modality alignment loss, and masked language modeling loss. The intra-modality alignment loss is used to ensure that the feature representation within each modality is consistent and discriminative. The loss is calculated by comparing the similarity of features within the same modality, such as using methods such as Euclidean distance or cosine similarity. The inter-modality alignment loss ensures that the feature representations between different modalities can effectively interact and fuse. The loss is calculated by comparing the similarity of features between different modalities, such as using methods such as cross entropy loss or mean squared error loss. The masked language modeling loss helps the model learn the dependencies and sequence information between features. The model can be trained by predicting the missing parts of the sequence, such as using the masked language modeling task of a pre-trained model such as BERT to calculate the loss.

[0126] In an implementation scenario, the goal of the joint loss function is to comprehensively consider the intra-modality alignment loss, inter-modality alignment loss, and masked language modeling loss to ensure that the model can simultaneously focus on the internal consistency of features, the correlation between different modalities, and the learning of sequence information during training. The intra-modality alignment loss, inter-modality alignment loss, and masked language modeling loss are combined with certain weights to form a joint loss function. The weights of different losses can be adjusted according to the nature of the task and the characteristics of the data to balance the impact of various losses on model training.

[0127] By defining a joint loss function and comprehensively considering the intra-modality alignment loss, inter-modality alignment loss, and masked language modeling loss, the model can be effectively trained to improve the accuracy and stability of classification. The design of this comprehensive loss function can more comprehensively consider information from multiple aspects, thereby improving the performance and generalization ability of the model on complex tasks.

[0128] In an implementation scenario, first, the fused features are mapped to the risk assessment space through a fully connected layer, and the formula is:

[0129] Z=H Fusion W FC +b FC

[0130] Among them, W FC ∈R d×c is the weight matrix of the fully connected layer, b FC ∈R C is the bias vector, and c is the number of categories.

[0131] Next, the output is passed through the softmax function to generate a probability distribution, which is used to represent the probability of belonging to each category. The specific formula is:

[0132]

[0133] Among them, P(y=j|H Fusion ) represents the predicted probability that the sample belongs to category j. Finally, the classification model is optimized by defining the cross entropy loss function, whose formula is:

[0134]

[0135] Among them, y ij is the one-hot encoding of the true category, and N is the number of training samples.

[0136] Through the above process, the multimodal fusion module and the classification module work together to successfully realize the full-process modeling from multimodal feature extraction to neonatal hypoxia risk prediction, improving the prediction accuracy and the overall performance of the model.

[0137] In other implementation scenarios, the classification module includes two interconnected fully connected layers and a Softmax activation function. The input data passes through the first fully connected layer, which linearly transforms the data and applies an activation function (such as ReLU) to introduce nonlinearity. The output of the first fully connected layer is used as the input of the second fully connected layer, which is linearly transformed and activated again. The first fully connected layer learns some abstract feature representations to map the input data to a higher-level feature space. The second fully connected layer further learns more complex and abstract features based on the first fully connected layer to better distinguish the differences between different categories. An output layer with a Softmax activation function is connected after the two fully connected layers. The Softmax function converts the output of the model into a probability distribution representing the probability of each category. Based on the probability distribution of the Softmax output, the probability of the input belonging to each category can be predicted, and the category with the highest probability can be selected as the final prediction result.

[0138] In other implementation scenarios, the fetal heart rate monitoring analysis model is trained and tested through training data. The public dataset in the training data comes from the Czech Technical University (CTU) and the University Hospital Brno (UHB), and contains 552 fetal heart rate monitoring (CTG) records collected between April 27, 2010 and August 6, 2012. Each record is up to 90 minutes long and contains fetal heart rate (FHR) and uterine contraction (UC) signal data, as well as detailed maternal and fetal clinical information, such as mother's age, fetal sex, mode of delivery, and delivery time. Please refer to Figure 6 and Figure 7 , Figure 6 and Figure 7 All are schematic diagrams of fetal heart rate and uterine contraction signal data. All recordings were approved by the Ethics Committee of the Brno University Hospital, and the participants signed informed consent.

[0139] The dataset consists of two parts: one is the fetal heart rate monitoring (CTG) data recorded during delivery, and the other is clinical data. All CTG data were collected using STAN S21, S31, Avalon FM40 and FM50 fetal monitors and were recorded by OB. The system converted the data into text format. After screening the clinical data (the screening criteria are described in the table of clinical data characteristics), 552 CTG records that met the clinical and technical requirements were finally identified, and the relevant clinical data were exported and converted into the Physionet text format.

[0140] CTG records and clinical data were processed anonymously by the hospital information system, and all time data were displayed anonymously relative to the date of birth. This was approved by the Ethics Committee of the Brno University Hospital. All participants signed an informed consent.

[0141] The private dataset of training data includes a screened positive control group, with a total of 409 cases. The inclusion of these cases was based on the following strict criteria: the Apgar score of the newborn at 1 minute (Apgar1) or 5 minutes (Apgar5) after birth, or the blood pH of the newborn was lower than 7.2. The Apgar score is a widely recognized measurement indicator for assessing the health status of newborns after birth. A score lower than or equal to 7 may indicate that the newborn has certain physiological challenges. In addition, the measurement of blood pH is a key indicator for assessing the acid-base balance status of the newborn. If the pH value is lower than 7.2, this may point to a potential pathological state in the newborn.

[0142] The clinical table data of the positive control group covers five dimensions: discharge records, medical records, ultrasound reports, delivery records, and neonatal records. The specific contents are shown in Table 3. The data forms of these features mainly include two types: first, the time series data records the key time nodes such as the maternal admission time in detail, providing a basis for dynamic analysis; second, all data features and corresponding numerical information are presented in a structured table form, making data processing and analysis more systematic and precise.

[0143]

[0144]

[0145] Table 3

[0146] The normal control group included a total of 2296 cases, and its inclusion criteria were strictly screened and filtered. Among the 12307 cases that underwent blood gas analysis after filtering, the cases that met the criteria of the positive control group were first excluded. Among the remaining cases, we further selected cases with blood pH values ​​between 7.35 and 7.45 as the normal control group. In addition, in order to ensure the accuracy of the data, the data with abnormal blood gas analysis detection time were excluded, and 2296 normal cases were finally determined. The data characteristics of the normal control group were consistent with those of the positive control group, covering five dimensions: discharge records, medical records, ultrasound reports, delivery records, and neonatal records. The data form mainly includes two types: time series and tabular data. The time series data recorded in detail the key time nodes such as maternal admission, delivery, fetal birth, blood gas analysis, and fetal heart monitoring. All data characteristics and corresponding numerical information are presented in a structured table form, making data processing and analysis more systematic and precise.

[0147] In this embodiment, the training data includes detailed fetal heart rate (CTG) data and clinical table data. Each case includes a fetal heart rate (FHR) signal and related demographic and clinical information, such as gender, gestational age, mode of delivery, duration of labor, maternal age, and previous delivery history. The data of each case is marked as high or low risk of hypoxia and then input into the fetal heart rate monitoring analysis model for training.

[0148] The Adam optimizer is used, and the training parameters are set as follows: 300 training cycles, 0.0001 learning rate, and 128 batch size. This parameter configuration ensures the stability and efficiency of the model during training.

[0149] In one embodiment, the performance of the fetal heart monitoring analysis model is evaluated through comparative experiments: (1) Pure text model: only the demographic information and labor records in the clinical table data are used for classification to predict the risk of neonatal hypoxia; (2) Pure visual model: only the CWT image features of fetal heart monitoring (FHR) are used for classification. The results of these two single modalities are compared with the multimodal fusion fetal heart monitoring analysis method proposed in the present invention to verify the improvement effect of multimodal information fusion on model performance. The evaluation indicators include accuracy, sensitivity, specificity and area under the ROC curve (AUC) to comprehensively evaluate the performance of the model in the task of predicting neonatal hypoxia. The comparison results are shown in Table 4:

[0150]

[0151]

[0152] Table 4

[0153] Compared with the single modality model, the multimodal fusion method of the present invention shows significant improvement. Compared with the pure visual model, the accuracy is improved by about 9.5%, and the AUC (area under the ROC curve) is improved by 0.082. The multimodal fusion model of the present invention can effectively integrate FHR images and clinical information, learn feature representations containing rich semantics and context, and thus achieve higher accuracy and reliability in hypoxia risk prediction.

[0154] From the above description, it can be seen that in this embodiment, the present invention combines the fetal heart rate signal and clinical table data, comprehensively utilizes different types of information sources, and enhances the understanding of the correlation between different information. The multimodal fusion method helps to consider various aspects of information more comprehensively, so that the model can more comprehensively and accurately evaluate the health status of the newborn, thereby improving the accuracy and reliability of the diagnosis.

[0155] The present invention also provides a monitoring device. The monitoring device includes a processor and a memory. The processor is coupled to the memory. A computer program is stored in the memory, and the processor executes the computer program when working to implement the above method. The detailed steps can be referred to above and will not be repeated here.

[0156] The present invention also provides a computer-readable storage medium. The computer-readable storage medium stores at least one computer program, which is used to be executed by a processor to implement the above method. The detailed steps can be referred to above and will not be repeated here. In one embodiment, the computer-readable storage medium can be a storage chip, a hard disk, or a mobile hard disk or a USB flash drive, an optical disk, or other readable and writable storage tools in a terminal, or a server, etc.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0158] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. For those skilled in the art, the technical solutions described in the above embodiments can be modified, or some of the technical features therein can be replaced by equivalents; and all these modifications and replacements should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A fetal heart monitoring analysis method, characterized in that: Applied to a fetal heart monitoring analysis model, the fetal heart monitoring analysis model comprising at least one set of interconnected ResNet50 networks and Mamba modules, a multi-scale text encoder, a cross-modal attention module, and a classification module; The fetal heart monitoring analysis method comprises: Obtain fetal heart rate signals and clinical table data; Performing continuous wavelet transform on the heart rate signal to obtain a time-frequency domain image, and inputting the time-frequency domain image into a ResNet50 network and a Mamba module to obtain image temporal features; Inputting the clinical table data into a multi-scale text encoder, obtaining dependency relationships in time series data of the clinical table data at different scales, and fusing the dependency relationships to obtain time series dependency features; A cross-modal attention mechanism is used to fuse the image temporal features and the temporal dependency features to obtain fused features; The fused features are input into a classification module to obtain a hypoxia risk assessment result, and a joint loss function of the classification module includes intra-modality alignment loss, inter-modality alignment loss and masked language modeling loss.

2. The fetal heart monitoring analysis method according to claim 1, characterized in that: The step of performing continuous wavelet transform on the heart rate signal to obtain a time-frequency domain image comprises: Mapping the heart rate signal to a two-dimensional time-frequency domain space, adjusting the frequency and time resolution by scaling and translating factors, and obtaining a two-dimensional matrix; The two-dimensional matrix is ​​normalized and pseudo-colored to be converted into the time-frequency domain image in RGB format.

3. The fetal heart monitoring analysis method according to claim 1, characterized in that: The step of inputting the time-frequency domain image into the ResNet50 network and the Mamba module to obtain the image time series features includes: The time-frequency domain image is adjusted to a preset size and then input into a ResNet50 network to obtain pattern and sequence features in the time-frequency domain image, and output a feature vector with a length of 2048; The feature vector is processed by combining the Mamba module with a multi-layer perceptron and a SiLU activation function, and the feature vector is converted and integrated into a target vector with a length of 512.

4. The fetal heart monitoring analysis method according to claim 1, characterized in that: The step of applying the multi-scale text encoder comprises: converting the clinical table data into a clinical data matrix; For each row of data in the clinical data matrix, one-dimensional convolution is used to extract local features to obtain local linear features and basic relationships between adjacent features; Using dilated convolution for the clinical data matrix to obtain long-range dependencies of different scales; Cascade the outputs of one-dimensional convolution and dilated convolution to obtain splicing features; Processing the concatenated features through an activation function and a fully connected layer to obtain the temporal dependency features; The time series dependent features are reduced in dimension to extract key information.

5. The fetal heart monitoring analysis method according to claim 4, characterized in that: The multi-scale text encoder includes a one-dimensional convolution module, a hole one-dimensional convolution module and a multi-layer perceptron connected in sequence; The one-dimensional convolution module includes a one-dimensional convolution block, a batch normalization and ReLU activation layer, and an average pooling layer connected in sequence; The atrous one-dimensional convolution module includes one-dimensional convolution blocks and batch normalization and ReLU activation layers connected in sequence; The convolution kernel size of the one-dimensional convolution block in the one-dimensional convolution module is k=3, the step size is s=i, and the padding is p=1; The convolution kernel size of the one-dimensional convolution block in the hole one-dimensional convolution module is k=3, the step size is s=i, the padding is p=1, and the expansion parameter is d=i; Among them, i is the number of layers of the one-dimensional convolutional block.

6. The fetal heart monitoring analysis method according to claim 1, characterized in that: The step of fusing the image temporal features and the temporal dependency features using a cross-modal attention mechanism comprises: Embedding the image temporal features and the temporal dependency features into a unified feature space to form a unified feature; The unified features are transformed linearly to generate a query matrix, a key matrix and a value matrix; The correlation among the query matrix, key matrix and value matrix is ​​calculated. Based on the correlation, the query, key and value are divided into multiple different heads through a multi-head attention mechanism. Each head calculates attention independently and is fused by splicing.

7. The fetal heart monitoring analysis method according to claim 1, characterized in that: The step of inputting the fusion features into a classification module comprises: Mapping the fusion features to the risk assessment space through a fully connected layer to obtain mapping features, inputting the mapping features into a softmax function to generate a probability distribution, and obtaining the probability of belonging to each health category; Before the step of inputting the fusion feature into the classification module, the method further includes: The classification module is optimized by defining a cross entropy loss function.

8. The fetal heart monitoring and analysis method according to any one of claims 1 to 7, characterized in that: The heart rate signal is recorded in the form of time series sequence data, including: time index, fetal heart rate at each quarter-second interval, uterine contraction intensity at each interval, seconds elapsed from the start of the monitoring period, and minutes elapsed from the start of the monitoring period; The clinical table data includes: clinical data of the delivery process used for excluding data, clinical data of the delivery process not involved in the exclusion but used for recording, clinical data of the delivery results, and fetal heart rate monitoring signal standards; The clinical data of the delivery process used to exclude data include: maternal age, gestational age, known fetal diseases, pregnancy type, and delivery type; The clinical data of the delivery process that were not included in the exclusion but were used for recording included: fetal sex, number of deliveries, risk factors, complications during pregnancy, saline-related indicators during pregnancy, medications, whether the delivery was induced, type of fetal position, stagnation during delivery, head-body incoordination, and cranio-pelvic incoordination; The clinical data of delivery outcomes described included: umbilical artery pH, partial carbon dioxide pressure, base excess, basal deficit in extracellular fluid, neonatal assessment, number of days in the neonatal intensive care unit, seizures, hypoxic-ischemic encephalopathy, intubation, and Apgar score; The fetal heart rate monitoring signal standards include: duration of the first stage of labor, duration of the second stage of labor, type of measuring equipment, and time interval from end of signal to delivery.

9. The fetal heart monitoring and analysis method according to any one of claims 1 to 7, characterized in that: The fetal heart monitoring analysis method further comprises: training the fetal heart monitoring analysis model through training data; The training data covers five dimensions: discharge records, medical records, ultrasound reports, delivery records, and neonatal records; the training data contains time series data, which records multiple key time nodes, including the time nodes of maternal admission, delivery, fetal birth, blood gas analysis, and fetal heart rate monitoring; The discharge records include: the Apgar score of the newborn at 1 minute after birth, the Apgar score of the newborn at 5 minutes after birth, the mother's age, weeks of pregnancy, the total number of pregnancies of the female, the total number of births of the female, whether the baby is macrosomia, whether the fetal growth is restricted, the baby's birth weight, gender, whether the newborn is asphyxia, whether the baby is anemic, whether the baby has gestational hypertension, whether the baby has diabetes, whether the baby has hyperthyroidism, whether the baby has hypothyroidism, and whether the baby has premature rupture of membranes; The medical records include: admission systolic blood pressure, admission diastolic blood pressure, presence or absence of fever, and whether induced labor was performed; The ultrasound report includes: amniotic fluid dark area, amniotic fluid index, umbilical blood flow systolic / diastolic ratio; The delivery record includes: the number of umbilical cord circles around the neck, the number of umbilical cord twists, whether there is a single umbilical artery, whether there is meconium in the amniotic fluid, fetal position, duration of the first stage of labor, stagnation during labor, duration of the second stage of labor, type of delivery, and type of intrapartum monitoring; The neonatal records include: umbilical artery blood pH value, umbilical artery blood base excess, umbilical artery standard bicarbonate concentration, and partial pressure of carbon dioxide in umbilical artery blood.

10. A monitoring device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Images