Examination data fusion method based on neural network model, eclampsia prediction method, computer equipment and computer readable storage medium

Abstract

The invention discloses an examination data fusion method based on a neural network model, an eclampsia prediction method and related equipment. Comprising the steps that inspection data of a target object are acquired, and the inspection data comprise multiple types of inspection modes and multiple types of inspection features under all the inspection modes; for at least part of the multiple types of inspection modes, performing mask processing on the multiple types of inspection features in the inspection modes through a first neural network sub-model in the neural network model according to a feature missing result in the inspection modes, and then performing fusion processing on the multiple types of inspection features subjected to mask processing, obtaining a feature fusion result under the inspection mode; and performing mask processing on the feature fusion result under the multi-type inspection modes according to the mode missing result through a second neural network sub-model in the neural network model, and performing fusion processing on the feature fusion result after mask processing to obtain a mode fusion result of the target object. The objective of the invention is to realize more accurate eclampsia prediction by effectively integrating examination data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an examination data fusion method based on a neural network model, a pre-eclampsia prediction method based on a neural network model, a computer device and a computer readable storage medium. BACKGROUND

[0002] At present, in clinical work, the prediction of some diseases is mainly realized according to the medical examination data of patients. In particular, the prediction of some diseases of pregnant women (such as pre-eclampsia, a disease occurring during pregnancy) needs to refer to some medical examination data of the pregnant women during pregnancy. It can be understood that the pregnant women will receive multiple direction examinations during pregnancy, such as urine, body fluid indicators, intraocular pressure, heart, fetal artery flow rate, etc. However, due to some special reasons, the same pregnant woman may not be examined in all examination modalities or all examination items under the examination modalities, resulting in the problem of inaccurate prediction results due to incomplete examination data when facing the prediction of some diseases.

[0003] Therefore, how to effectively integrate the existing medical examination data of patients for reasonable use and further improve the accuracy of pre-eclampsia prediction is a problem to be solved by those skilled in the art. SUMMARY

[0004] The purpose of the present application is to provide an examination data fusion method based on a neural network model, which can effectively integrate the existing medical examination data of patients for reasonable use, and help to improve the accuracy of pre-eclampsia prediction using the fused examination data. Another purpose of the present application is to provide a pre-eclampsia prediction method based on a neural network model, a computer device and a computer readable storage medium, all of which have the above beneficial effects.

[0005] In a first aspect, the present application discloses an examination data fusion method based on a neural network model, comprising:

[0006] obtaining examination data of a target object; wherein the examination data comprises multiple types of examination modalities and multiple types of examination features under each examination modality;

[0007] For at least part of the multiple types of examination modalities, a first neural network sub-model in the neural network model is used to first mask process the multiple types of examination features under the examination modality according to the feature missing result under the examination modality, and then fuse process the multiple types of examination features after the mask processing, to obtain a feature fusion result under the examination modality;

[0008] The second neural network sub-model in the neural network model first performs mask processing on the feature fusion result under the multiple types of examination modalities according to the modality missing result, and then performs fusion processing on the mask-processed feature fusion result to obtain the modality fusion result of the target object.

[0009] Optionally, in a case where the multiple types of examination features under the examination modality are examination features of the examination modality in multiple different examination periods, after the examination data of the target object is acquired, the method further includes:

[0010] For at least part of the multiple types of examination modalities, multiple examination features of the same type in multiple different examination periods are fused to obtain a fusion result of the examination feature of the corresponding type, and the fusion result is taken as the multiple types of examination features under the examination modality.

[0011] Optionally, the fusion of the multiple examination features of the same type in multiple different examination periods to obtain the fusion result of the examination feature of the corresponding type includes:

[0012] The weight values of the multiple examination features of the same type in the multiple examination periods are determined according to a time sequence of the multiple examination periods; and the weight values increase from far to near according to the time sequence of the examination periods.

[0013] The multiple examination features of the same type in the multiple examination periods are weighted and summed according to the weight values to obtain the fusion result of the examination feature of the corresponding type.

[0014] Optionally, the examination data is pregnancy examination data of the target object.

[0015] Correspondingly, the examination periods include gestational weeks and gestational days.

[0016] Optionally, the first neural network sub-model and the second neural network sub-model each include a mask network and an attention network.

[0017] Optionally, before the mask processing of the multiple types of examination features under the examination modality according to the feature missing result under the examination modality, the method further includes:

[0018] The missing examination feature type under the examination modality is determined according to a preset examination feature type under the examination modality and the multiple types of examination features under the examination modality, so as to take the missing examination feature type under the examination modality as the feature missing result under the examination modality.

[0019] Correspondingly, the mask processing of the multiple types of examination features under the examination modality according to the feature missing result under the examination modality includes:

[0020] Using the masking network, a feature missing vector for the inspection mode is first constructed based on the multi-type inspection features under the inspection mode and the feature missing result. Then, the multi-type inspection features under the inspection mode are masked using the feature missing vector to obtain the masked multi-type inspection features.

[0021] Optionally, the fusion processing of the multi-type inspection features after masking to obtain the feature fusion result under the inspection modality includes:

[0022] Using the attention network, the first learning matrix is ​​first updated based on the missing feature vector, and then the updated first learning matrix is ​​used to fuse the multi-type inspection features after the masking process to obtain the feature fusion result under the inspection modality.

[0023] Optionally, before performing masking processing on the feature fusion results under the multi-type inspection modality based on the modality missing results, the method further includes:

[0024] The missing inspection modality type is determined based on the preset inspection modality type and the multiple inspection modal types, and the missing inspection modality type is used as the modality missing result;

[0025] Accordingly, the feature fusion results under the multi-type inspection modalities are masked based on the modality missing results, including:

[0026] Using the masking network, a mode missing vector is first constructed based on the multi-type inspection mode and the mode missing result. Then, the mode missing vector is used to mask the feature fusion result under the multi-type inspection mode to obtain the masked feature fusion result.

[0027] Optionally, the step of performing fusion processing on the feature fusion result after masking to obtain the modality fusion result of the target object includes:

[0028] Using the attention network, the second learning matrix is ​​first updated according to the modality missing vector, and then the updated second learning matrix is ​​used to fuse the feature fusion result after the masking process to obtain the modality fusion result of the target object.

[0029] Optionally, both the first neural network sub-model and the second neural network sub-model further include a fully connected network.

[0030] Optionally, after fusing the multi-type inspection features after masking to obtain the feature fusion result under the inspection modality, the method further includes:

[0031] The full connection network is used to compress the feature fusion result under the examination modality according to a first feature size, to obtain a feature compression result under the examination modality.

[0032] Optionally, after the fusion processing of the feature fusion result after the mask processing is performed, the modality fusion result of the target object is obtained, and the method further includes:

[0033] The full connection network is used to compress the modality fusion result of the target object according to a second feature size, to obtain a modality compression result of the target object.

[0034] In a second aspect, the present application discloses a method for predicting eclampsia based on a neural network model, including:

[0035] Obtaining examination data of a target object; wherein the examination data includes multiple types of examination modalities and multiple types of examination features under each examination modality;

[0036] For at least part of the multiple types of examination modalities, a first neural network sub-model in the neural network model is used to first perform mask processing on the multiple types of examination features under the examination modality according to the feature missing result under the examination modality, and then perform fusion processing on the multiple types of examination features after the mask processing, to obtain a feature fusion result under the examination modality;

[0037] A second neural network sub-model in the neural network model is used to first perform mask processing on the feature fusion result under the multiple types of examination modalities according to the modality missing result, and then perform fusion processing on the feature fusion result after the mask processing, to obtain a modality fusion result of the target object;

[0038] A prediction network in the neural network model is used to process the modality fusion result, to obtain an eclampsia prediction result of the target object.

[0039] In a third aspect, the present application discloses a computer device, including:

[0040] A memory for storing a computer program;

[0041] A processor for executing the computer program to implement the steps of any one of the examination data fusion methods or the steps of the eclampsia prediction method.

[0042] In a fourth aspect, the present application discloses a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the examination data fusion methods or the steps of the eclampsia prediction method.

[0043] The application provides an examination data fusion method based on a neural network model, comprising: acquiring examination data of a target object; wherein the examination data comprises multiple types of examination modalities and multiple types of examination features under each examination modality; for at least part of the multiple types of examination modalities, a first neural network sub-model in the neural network model is used to first perform mask processing on the multiple types of examination features under the examination modality according to a feature missing result under the examination modality, and then perform fusion processing on the multiple types of examination features after the mask processing, to obtain a feature fusion result under the examination modality; a second neural network sub-model in the neural network model is used to first perform mask processing on the feature fusion result under the multiple types of examination modalities according to a modality missing result, and then perform fusion processing on the feature fusion result after the mask processing, to obtain a modality fusion result of the target object.

[0044] The application provides an examination data fusion method based on a neural network model, comprising: acquiring examination data of a target object; wherein the examination data comprises multiple types of examination modalities and multiple types of examination features under each examination modality; for at least part of the multiple types of examination modalities, a first neural network sub-model in the neural network model is used to first perform mask processing on the multiple types of examination features under the examination modality according to a feature missing result under the examination modality, and then perform fusion processing on the multiple types of examination features after the mask processing, to obtain a feature fusion result under the examination modality; a second neural network sub-model in the neural network model is used to first perform mask processing on the feature fusion result under the multiple types of examination modalities according to a modality missing result, and then perform fusion processing on the feature fusion result after the mask processing, to obtain a modality fusion result of the target object.

[0045] The application provides an examination data fusion method based on a neural network model, comprising: acquiring examination data of a target object; wherein the examination data comprises multiple types of examination modalities and multiple types of examination features under each examination modality; for at least part of the multiple types of examination modalities, a first neural network sub-model in the neural network model is used to first perform mask processing on the multiple types of examination features under the examination modality according to a feature missing result under the examination modality, and then perform fusion processing on the multiple types of examination features after the mask processing, to obtain a feature fusion result under the examination modality; a second neural network sub-model in the neural network model is used to first perform mask processing on the feature fusion result under the multiple types of examination modalities according to a modality missing result, and then perform fusion processing on the feature fusion result after the mask processing, to obtain a modality fusion result of the target object.

[0046] The application provides a computer device and a computer readable storage medium, which also have the above technical effects, and the application will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the prior art and the technical solutions in the embodiments of the present application, the drawings used in the description of the prior art and the embodiments of the present application will be briefly introduced. Of course, the drawings described below in the description of the embodiments of the present application are only a part of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings provided without creative labor, and the obtained other drawings also belong to the protection scope of the present application.

[0048] Figure 1 A flowchart of a check data fusion method based on a neural network model provided by an embodiment of the present application;

[0049] Figure 2 A flowchart of a check data vectorization processing provided by an embodiment of the present application;

[0050] Figure 3 A network architecture diagram of a first neural network sub-model provided by an embodiment of the present application;

[0051] Figure 4 A network architecture diagram of a second neural network sub-model provided by an embodiment of the present application;

[0052] Figure 5 A network architecture diagram of a neural network model provided by an embodiment of the present application;

[0053] Figure 6 A flowchart of a pre-eclampsia prediction method based on a neural network model provided by an embodiment of the present application;

[0054] Figure 7 A general framework diagram of a check data fusion method for implementing diagnosis of pre-eclampsia provided by an embodiment of the present application;

[0055] Figure 8 A flowchart of a multi-feature data fusion method provided by an embodiment of the present application;

[0056] Figure 9 A flowchart of a multi-modal data fusion method provided by an embodiment of the present application;

[0057] Figure 10 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The core of the present application is to provide an examination data fusion method which can effectively integrate medical examination data of a patient for reasonable utilization, and help improve the accuracy of predicting eclampsia using the medical examination data after fusion processing. Another core of the present application is to provide an eclampsia prediction method based on a neural network model, a computer device and a computer readable storage medium, all of which have the above beneficial effects.

[0059] In order to more clearly and completely describe the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0060] The present application provides an examination data fusion method.

[0061] Please refer to Figure 1 , Figure 1 The flowchart of the examination data fusion method based on a neural network model provided by the embodiments of the present application can include but is not limited to S101-S103.

[0062] S101: Obtain examination data of a target object; wherein the examination data includes multiple types of examination modalities and multiple types of examination features under each examination modality.

[0063] This step aims to realize the acquisition of the examination data of the target object. The target object is the current diagnosis object, such as a pregnant woman; the examination data can include multiple types of examination modalities and multiple types of examination features under each type of examination modality. The examination modality refers to different types of examination modalities for the target object, such as blood detection, uterus detection, liver function detection, etc.; the examination feature refers to the examination feature index under a specific examination modality, such as hemoglobin, red blood cell count, white blood cell count, platelet count, etc. under the blood detection modality, uterine artery pressure pulsation index, resistance index, etc. under the uterus detection modality, and transaminase, bilirubin, albumin, etc. under the liver function detection modality.

[0064] In an embodiment of the present application, when the multiple types of examination features under the examination modality are the examination features of the examination modality in multiple different examination cycles, after obtaining the examination data of the target object, it can further include: for at least part of the multiple types of examination modalities, fusing multiple examination features of the same type in multiple different examination cycles to obtain the fusion result of the examination features of the corresponding type, and taking it as the multiple types of examination features under the examination modality.

[0065] As described above, for predicting certain diseases of a pregnant woman, some medical examination data of the pregnant woman in a pregnancy period needs to be referred to, and the pregnancy period is a relatively long time period. In order to effectively ensure the accuracy of subsequent medical diagnosis, time information, i.e., the above examination period, can be added to the examination data, so as to implement data fusion processing based on the examination data of the target object in different examination periods and then complete subsequent medical diagnosis. Based on this, the multi-type examination features under the examination modality can be the examination features of the examination modality in multiple different examination periods. On this basis, after obtaining the examination data of the target object, the examination features of the same feature category in different examination periods can be fused before multi-feature fusion, i.e., under each examination modality, the examination features of the same feature type can be fused according to the time sequence of each examination period, to obtain the feature fusion result of each feature type under the corresponding examination modality, and the feature fusion result is taken as the multi-type examination feature under the corresponding examination modality.

[0066] For example, taking the examination feature "hemoglobin" under the blood test modality as an example, it is assumed that the examination data contains the hemoglobin detection results of the sixth, eighth and twelfth gestational weeks. Then, the three hemoglobin detection results can be fused (e.g., weighted summation) according to the time sequence of the three examination periods, to obtain the hemoglobin feature fusion result. Then, after determining the missing features under the blood test modality, the hemoglobin feature fusion result can be fused with other feature fusion results (e.g., red blood cell count feature fusion result, white blood cell count feature fusion result, and platelet count feature fusion result) under the blood test modality based on the feature missing result, to obtain the multi-feature fusion result under the blood test modality, i.e., the multi-type examination feature under the blood test modality.

[0067] In addition, to further ensure the accuracy of the time characteristics, when the examination data is the pregnancy examination data of the target object, the examination period can include the gestational weeks and the gestational days. It can be understood that, in the aspect of maternal information collection, different clinical examination modalities (such as ultrasonic examination, blood biochemistry, fetal heart monitoring, etc.) often collect data according to a specific gestational week, and the traditional data standardization method usually adopts a normalization strategy of “detection value, gestational day median”. However, this method has three significant defects: (1) a large number of samples are needed to calculate a reliable median benchmark; (2) the dynamic calculation of the gestational day median has high complexity; and (3) the long-term maintenance and update cost is large. To solve these problems, the embodiment of the present application introduces the time encoding mechanism of the Informer time prediction model, and constructs a two-dimensional gestational week representation system: the gestational period is divided into gestational weeks (1-41 weeks) and gestational days (0-6 days) two orthogonal characteristics, which not only retains the macro-pregnancy stage characteristics, but also realizes the day-level time resolution. Therefore, the accuracy of the examination period can be effectively ensured, and on this basis, the examination characteristics of the same feature type are fused according to the time sequence of each examination period to obtain the feature fusion result of each feature type under the examination modality, which can further ensure the accuracy of the feature fusion result, thereby effectively ensuring the accuracy of the final examination data fusion result.

[0068] In the specific implementation process, please refer to Figure 2 , Figure 2 The flowchart of the examination data vectorization processing provided by the embodiment of the present application is as follows: for each examination feature under any examination modality, the examination value, gestational week and gestational day corresponding to the examination feature can be respectively vectorized to obtain an examination value vector, a gestational week vector and a gestational day vector, and then the three vectors are added to obtain an examination feature vector fused with time encoding. It can be understood that, in the process of data processing based on a neural network, the data to be processed is usually vectorized into a 256-dimensional vector or a 512-dimensional vector for data processing. In the embodiment of the present application, each examination feature vector can be a 256-dimensional examination feature vector.

[0069] Further, a plurality of examination characteristics of the same type in a plurality of different examination periods are fused to obtain a fusion result of the examination characteristics of the corresponding type, including: determining weight values of a plurality of examination characteristics of the same type in a plurality of examination periods according to the time sequence of the plurality of examination periods; the weight values increase from far to near according to the time sequence of the examination periods; and the plurality of examination characteristics of the same type in the plurality of examination periods are weighted and summed according to the weight values to obtain the fusion result of the examination characteristics of the corresponding type.

[0070] The embodiment of the application provides an implementation method for fusing inspection features of the same feature type according to the time sequence of each inspection period. Specifically, for the same inspection features of different inspection periods, a weight setting mode that the closer the time, the greater the weight, and the farther the time, the smaller the weight can be used to assign weight values to the same inspection features in each inspection period, and then the fusion processing of the inspection features of the same feature type is realized by referring to the weight values of the inspection features.

[0071] For example, the default weight value 0.6 can be set in advance as the weight value of the inspection feature of the nearest inspection period. Assuming that there are three inspection features of the same feature type of different inspection periods, the weight value of the same inspection feature of the second nearest inspection period can be (1-0.6) * 0.6, and the weight value of the same inspection feature of the third nearest inspection period can be 1-(1-0.6) * 0.6-0.6. Taking the inspection feature "hemoglobin" in the blood detection mode as an example, the corresponding feature fusion result (hemoglobin feature fusion vector) = nearest inspection period hemoglobin inspection vector * 0.6 + second nearest inspection period hemoglobin inspection vector * [(1-0.6) * 0.6] + third nearest inspection period hemoglobin inspection vector * [1-(1-0.6) * 0.6-0.6]. Further, other inspection features (such as red blood cells, white blood cells, and platelets) in the blood detection mode are processed in this way to obtain their respective feature fusion results (corresponding red blood cell count feature fusion result / vector, white blood cell count feature fusion result / vector, and platelet count feature fusion result / vector). It can be understood that, in combination with the time coding mechanism described above, on the basis of effectively ensuring the accuracy of the inspection period, when the weight distribution is performed on the inspection features of the same feature type according to the time sequence of each inspection period, more appropriate and accurate weight distribution can be ensured, and the accuracy of the corresponding feature fusion result is ensured, thereby effectively ensuring the accuracy of the final inspection data fusion result.

[0072] S102: For at least part of the multiple types of inspection modalities, the first neural network sub-model in the neural network model is used to first mask the multiple types of inspection features under the inspection modality according to the feature missing result under the inspection modality, and then fuse the multiple types of inspection features after the mask processing to obtain the feature fusion result under the inspection modality.

[0073] The step is designed to realize multi-type feature fusion processing under the same inspection mode. Specifically, for at least part of the inspection modes in the multi-type inspection mode, the first neural network sub-model in the pre-created neural network model can be used to refer to the feature missing results under each inspection mode, and the multi-type inspection features under each inspection mode are sequentially subjected to mask processing and fusion processing, so as to obtain the feature fusion results under each inspection mode. Thus, the multi-type feature fusion processing based on missing features under the same inspection mode is realized.

[0074] In a possible implementation, the first neural network sub-model can specifically include a mask network and an attention network, wherein the mask network can be used to realize the multi-type feature mask processing under each inspection mode, and the attention network is used to realize the multi-type feature fusion processing under each inspection mode.

[0075] In an embodiment of the present application, before the multi-type inspection features under the inspection mode are subjected to mask processing according to the feature missing results under the inspection mode, it can further include: determining the missing inspection feature types under the inspection mode according to the preset inspection feature types under the inspection mode and the multi-type inspection features under the inspection mode, so as to take the missing inspection feature types under the inspection mode as the feature missing results under the inspection mode.

[0076] Correspondingly, the mask processing of the multi-type inspection features under the inspection mode according to the feature missing results under the inspection mode can include: using the mask network to first construct a feature missing vector under the inspection mode according to the multi-type inspection features under the inspection mode and the feature missing results, and then using the feature missing vector to mask process the multi-type inspection features under the inspection mode to obtain the multi-type inspection features after mask processing.

[0077] Specifically, the type of examination feature that needs to be collected under the current examination modality can be determined first, i.e., a preset examination feature type, and then the types of examination features actually obtained under the current examination modality are compared with the preset examination feature type, i.e., a missing examination feature type under the current examination modality, i.e., a feature missing result under the current examination modality. Thus, a feature missing vector under the current examination modality can be constructed according to the types of examination features and the feature missing result under the current examination modality. Taking the blood detection modality as an example, the blood detection modality should originally include hemoglobin, red blood cell count, white blood cell count, and platelet count. Assuming that the types of examination features actually obtained under the blood detection modality include hemoglobin, red blood cell count, and platelet count, it can be determined through comparison that the white blood cell count is a missing feature under the blood detection modality, and thus a feature missing vector under the blood detection modality [1, 1, 0, 1] can be constructed. Further, the types of examination features under the current examination modality can be masked by using the feature missing vector, and thus the types of examination features under the current examination modality after the masking can be obtained. In the masking process, all examination feature vectors under the current examination modality can be spliced based on the examination feature vectors fused with time encoding to obtain an initial feature splicing vector. Taking each examination feature vector as a 256-dimensional examination feature vector as an example, N (N represents the number of preset examination features under the current examination modality, and N has different values for different examination modalities) examination feature vectors can be spliced to obtain an initial feature splicing vector with a dimension of N x 256. Thus, the initial feature splicing vector can be masked by referring to the feature missing vector, and thus a feature splicing vector after the masking can be obtained, which is essentially a feature splicing vector with missing feature vectors set to 0 in the initial feature splicing vector. It can be understood that the above masking process can be implemented based on a masking network in the first neural network submodel.

[0078] On this basis, the types of examination features after the masking can be fused to obtain a feature fusion result under the examination modality, which can include: first updating a first learning matrix according to the feature missing vector by using an attention network, and then fusing the types of examination features after the masking by using the updated first learning matrix to obtain the feature fusion result under the examination modality.

[0079] Specifically, in the multi-type feature fusion process of the current examination modality, the first learning matrix, i.e., the learning matrix of the attention network in the first neural network sub-model, can be updated according to the feature missing vector. The first learning matrix essentially represents the weight information of each type of examination feature under the current examination modality. In combination with the feature missing vector, the first learning matrix represents the feature missing situation under the current examination modality. Therefore, the first learning matrix can be updated according to the feature missing vector. Specifically, for the missing features under the current examination modality, a lower weight value can be assigned in the first learning matrix, while for each type of feature actually collected under the current examination modality, a higher weight value can be assigned in the first learning matrix. At the same time, in the process of updating the first learning matrix, the historical feature missing situation under the current examination modality can also be referred to, i.e., for the examination features that are frequently missing under the current examination modality, a lower weight value can be assigned in the first learning matrix, for the examination features that are rarely missing under the current examination modality, a medium weight value can be assigned in the first learning matrix, and for the examination features that can always be collected under the current examination modality, a higher weight value can be assigned in the first learning matrix. Of course, regardless of how the weight values of each examination feature in the first learning matrix change, the sum of all weight values must be 1.

[0080] Further, the multi-type examination features processed by the mask are fused using the updated first learning matrix to obtain the feature fusion result under the current examination modality. In the feature fusion process, first, all examination feature vectors under the current examination modality can be stacked to obtain an initial examination feature matrix N×256 under the current examination modality, where N represents the number of preset examination features under the current examination modality (N takes different values for different examination modalities), and 256 represents that each examination feature vector is a 256-dimensional feature vector. Then, the feature fusion matrix (i.e., the feature fusion result) under the current examination modality can be calculated according to the updated first learning matrix and the initial examination feature matrix under the current examination modality. The size of the matrix is also N×256. It can be envisaged that the attention network can use a multi-head attention network to improve the operation efficiency through multi-head parallel operation, thereby ensuring the data fusion efficiency.

[0081] In addition, the first neural network sub-model can also include a fully connected network. On this basis, after the multi-type examination features processed by the mask are fused to obtain the feature fusion result under the examination modality, the feature fusion result under the examination modality can be compressed according to the first feature size using the fully connected network to obtain the feature compression result under the examination modality.

[0082] As described above, different N values correspond to different examination modalities, and based on this, the data size can be unified, that is, the feature fusion result under each examination modality is compressed according to the first feature size, so that in the subsequent S103, the multi-modal fusion can be performed according to the modality missing result to fuse all feature compression results, and the modality fusion result of the target object is obtained. The first feature size can be set according to actual conditions, which is not limited in the present application. For example, the first feature size can be 8x256, that is, the feature fusion matrix Nx256 under each detection modality can be converted into a feature compression matrix 8x256 (i.e. feature compression result) through a fully connected network.

[0083] In summary of the above embodiments, it can be referred to the above description of the first embodiment. Figure 3 , Figure 3 A network architecture diagram of a first neural network sub-model provided by an embodiment of the present application, the first neural network sub-model including a mask network (mask attention layer), an attention network (multi-head attention layer), and a fully connected network (fully connected layer). The input of the mask network is the splicing result of all examination feature vectors under the current examination modality, and the output is a feature splicing vector with the missing feature vectors in the splicing result set to 0. The input of the attention network is the output of the mask network, and the output is a feature fusion matrix (feature fusion result) under the current examination modality. The input of the fully connected network is the output of the attention network, and the output is a feature compression matrix (feature compression result) under the current examination modality.

[0084] S103: Through the second neural network sub-model in the neural network model, the feature fusion results under multiple types of examination modalities are first masked according to the modality missing result, and then the feature fusion results after the mask processing are fused to obtain the modality fusion result of the target object.

[0085] This step aims to realize multi-modal fusion processing. Specifically, after obtaining the feature fusion results under each examination modality, the second neural network sub-model in the pre-created neural network model can be used to sequentially perform mask processing and fusion processing on all current examination modalities according to the modality missing result, thereby obtaining the modality fusion result of the target object. Thus, multi-modal fusion processing based on missing modalities is realized, that is, the final fusion processing of the target object examination data is realized.

[0086] It can be understood that in the above examination data fusion process, the missing features and missing modalities are comprehensively considered, and in the subsequent step, the target object is subjected to corresponding medical diagnosis based on the multi-modal fusion result, which can effectively ensure the accuracy of the final medical examination result.

[0087] In a possible implementation, the second neural network sub-model can also be a neural network model based on a mask network and an attention network, where the mask network can be used to implement mask processing of the feature fusion result under the multiple types of inspection modalities, and the attention network can be used to implement fusion processing of the feature fusion result under the multiple types of inspection modalities.

[0088] In an embodiment of the present application, before the feature fusion result under the multiple types of inspection modalities is subjected to mask processing according to the modality missing result, the method can further include: determining a missing inspection modality type according to the preset inspection modality type and the multiple types of inspection modalities, so as to take the missing inspection modality type as the modality missing result.

[0089] Correspondingly, the mask processing of the feature fusion result under the multiple types of inspection modalities according to the modality missing result can include: first constructing a modality missing vector according to the multiple types of inspection modalities and the modality missing result by using a mask network, and then using the modality missing vector to perform mask processing on the feature fusion result under the multiple types of inspection modalities to obtain a feature fusion result after mask processing.

[0090] Specifically, the inspection modality type that needs to be collected for the target object's current inspection item itself, i.e., the preset inspection modality type, can be determined first, and then the current actually acquired multiple types of inspection modalities are compared with the preset inspection modality type, so as to determine the missing modality, i.e., the modality missing result. Thus, the current modality missing vector can be constructed according to the current actually acquired multiple types of inspection modalities and the modality missing result. Taking the preset inspection modality type that should originally include blood detection modality, uterus detection modality, and liver function detection modality as an example, assuming that the currently actually acquired inspection modalities include blood detection modality and uterus detection modality, it can be determined through comparison that the liver function detection modality is the missing modality, and thus the modality missing vector [1, 1, 0] can be constructed. Further, the feature fusion result under the current multiple types of inspection modalities can be subjected to mask processing by using the modality missing vector, so as to obtain the feature fusion result after mask processing. In the mask processing process, on the basis of the feature compression matrix 8x256 under the current each inspection modality, the feature compression matrix 8x256 under each inspection modality can be first stretched into a feature compression vector of 2048 (8x256) dimensions, and then the feature compression vectors under the current multiple types of inspection modalities can be stacked, so as to obtain an initial modality splicing matrix Mx2048, where M represents the number of preset inspection modality types. Thus, the initial modality splicing matrix Mx2048 can be subjected to mask processing by referring to the modality missing vector, so as to obtain the feature fusion result after mask processing, which is essentially a modality splicing matrix after the feature compression vector corresponding to the missing modality in the initial modality splicing matrix Mx2048 is set to 0. It can be understood that the above mask processing process can be implemented based on the mask network in the second neural network sub-model.

[0091] On this basis, the mask-processed feature fusion result is fused to obtain a modal fusion result of the target object, which can include: using an attention network, first updating a second learning matrix according to the modal missing vector, and then using the updated second learning matrix to fuse the mask-processed feature fusion result to obtain the modal fusion result of the target object.

[0092] Specifically, in the multi-modal fusion process, the second learning matrix, i.e., the learning matrix of the attention network in the second neural network sub-model, can be updated according to the modal missing vector. The second learning matrix essentially represents the weight information of each type of examination modality, and in combination with the modal missing vector, it represents the modal missing situation under the current examination item. Therefore, the second learning matrix can be updated according to the modal missing vector. Specifically, for the missing modal under the current examination item, a lower weight value can be assigned in the second learning matrix, and for each type of examination modality actually collected under the current examination item, a higher weight value can be assigned in the second learning matrix. At the same time, in the process of updating the second learning matrix, the historical modal missing situation under the current examination item can also be referred to, i.e., for the examination modal that is frequently missing under the current examination item, a lower weight value can be assigned in the second learning matrix, for the examination modal that is infrequently missing under the current examination item, a medium weight value can be assigned in the second learning matrix, and for the examination modal that can always be collected under the current examination item, a higher weight value can be assigned in the second learning matrix. Of course, regardless of how the weight values of each examination modality in the second learning matrix change, the sum of all weight values must be 1.

[0093] Further, the mask-processed feature fusion result is fused using the updated second learning matrix to obtain the modal fusion result of the target object. In the multi-modal fusion process, the modal fusion matrix (modal fusion result) of the target object can be calculated according to the updated second learning matrix and the above modal concatenation matrix, and of course, the size of the matrix is also Mx2048. Similarly, the attention network can use a multi-head attention network to improve the operation efficiency through multi-head parallel operation, thereby ensuring the efficiency of the examination data fusion.

[0094] In addition, the second neural network sub-model can also include a fully connected network; after the mask-processed feature fusion result is fused to obtain the modal fusion result of the target object, the fully connected network can be used to compress the modal fusion result of the target object according to the second feature size to obtain a modal compression result of the target object.

[0095] Similarly, to facilitate subsequent data calculations, the modality compression result can be compressed. This process can be achieved using the fully connected network in the second neural network sub-model with reference to the second feature size. The second feature size can be set according to the actual situation, and this application does not limit it. For example, the second feature size can be 6×64, that is, the modality fusion matrix M×2048 can be converted into a modality compression matrix 6×64 (i.e., the feature compression result) through the fully connected network.

[0096] Based on the above embodiments, reference can be made to the following: Figure 4 , Figure 4 This is a network architecture diagram of a second neural network sub-model provided in an embodiment of this application. The second neural network sub-model includes a mask network (masked attention layer), an attention network (multi-head attention layer), and a fully connected network (fully connected layer). The input to the mask network is an initial modality concatenation matrix obtained by stacking feature compression vectors under the current multi-type inspection modality, and the output is a modality concatenation matrix after setting the feature compression vectors corresponding to the missing modalities in the initial modality concatenation matrix to 0. The input to the attention network is the output of the mask network, and the output is the modality fusion matrix of the target object (modality fusion result). The input to the fully connected network is the output of the attention network, and the output is the modality compression matrix (modality compression result).

[0097] Combination Figure 3 and Figure 4 Please refer to Figure 5 , Figure 5 This is a network architecture diagram of a neural network model provided in an embodiment of this application. The neural network model includes... Figure 3 The first neural network sub-model shown combines Figure 4 The second neural network sub-model is shown. It is understandable that the neural network model can be pre-trained based on data samples and can be directly called upon during use. During model training, an initial neural network model can be built based on mask networks, attention networks, and fully connected networks. Then, this initial neural network model is iteratively trained using inspection feature data samples under various inspection modalities until the model converges to obtain the final neural network model.

[0098] It can be seen that the examination data fusion method based on the neural network model provided in the embodiments of the present application first acquires the examination data of the target object, including multiple types of examination modalities and multiple types of examination features under each examination modality. Then, for at least part of the multiple types of examination modalities, the first neural network sub-model in the pre-created neural network model is used to refer to the feature missing result under each examination modality to perform data fusion processing on the multiple types of examination features under the corresponding examination modality, so as to realize the multi-feature fusion processing based on the missing features under the same examination modality. Finally, the second neural network sub-model in the pre-created neural network model is used to refer to the modality missing result to perform data fusion processing on the feature fusion result under the multiple types of examination modalities, so as to realize the multi-modality fusion processing based on the missing modalities. It can be seen that the technical solution realizes the effective integration of the examination data of the target object based on the missing data by using the neural network model, further improves the utilization rate of the medical examination data, that is, realizes the effective utilization of the medical examination data, and helps to effectively improve the accuracy of predicting eclampsia by using the medical examination data.

[0099] The embodiments of the present application provide a neural network model-based eclampsia prediction method.

[0100] Please refer to Figure 6 , Figure 6 The flowchart of the neural network model-based eclampsia prediction method provided in the embodiments of the present application can include but is not limited to the following S201-S204.

[0101] S201: Acquire the examination data of the target object; wherein the examination data includes multiple types of examination modalities and multiple types of examination features under each examination modality;

[0102] S202: For at least part of the multiple types of examination modalities, the first neural network sub-model in the neural network model is used to first perform mask processing on the multiple types of examination features under the examination modality according to the feature missing result under the examination modality, and then perform fusion processing on the mask-processed multiple types of examination features, to obtain the feature fusion result under the examination modality;

[0103] S203: The second neural network sub-model in the neural network model is used to first perform mask processing on the feature fusion result under the multiple types of examination modalities according to the modality missing result, and then perform fusion processing on the mask-processed feature fusion result, to obtain the modality fusion result of the target object;

[0104] S204: The prediction network in the neural network model is used to process the modality fusion result, to obtain the eclampsia prediction result of the target object.

[0105] Specifically, the examination data fusion method provided in the foregoing embodiments can be applied to preeclampsia prediction, i.e., connecting a prediction network after the first neural network submodel and the second neural network submodel, for processing the modality fusion result output by the second neural network submodel to obtain a preeclampsia prediction result of the target object. Obviously, this process is essentially to implement preeclampsia prediction of the target object based on the examination data fusion result of the target object to predict the probability of the patient subsequently suffering from preeclampsia.

[0106] In a possible implementation, the prediction network can be a fully connected network. It can be understood that preeclampsia prediction is generally divided into three different period prediction stages (such as 32 weeks of pregnancy, 36 weeks of pregnancy, and 38 weeks of pregnancy). Therefore, on the basis of obtaining the modality compression matrix based on the second neural network submodel, the 6x64 modality compression matrix can be converted into a new modality compression matrix 3x2 (modality transformation matrix) through the fully connected network. At this time, the modality transformation matrix 3x2 can be split into 3 two-dimensional vectors, and each two-dimensional vector corresponds to a prediction stage. Thus, the preeclampsia prediction result of the corresponding prediction stage can be determined according to the prediction result of each two-dimensional vector.

[0107] In the specific implementation process, after obtaining the modality transformation matrix 3x2 of the target object, for each row element in the modality transformation matrix 3x2, the current two elements (corresponding to two columns) in the row are first mapped to the interval (0, 1) by using the sigmnoid function, and then the two transformed elements in the current row are compared in value to obtain the prediction result of the preeclampsia prediction stage corresponding to the current row. That is:

[0108] (1) The transformed element in the first column of the first row is compared in value with the transformed element in the second column of the first row. If the transformed element in the first column of the first row is greater than the transformed element in the second column of the first row in value, it is determined that the target object will not suffer from preeclampsia in the first preeclampsia prediction stage (such as 32 weeks of pregnancy). If the transformed element in the first column of the first row is less than the transformed element in the second column of the first row in value, it is determined that the target object will suffer from preeclampsia in the first preeclampsia prediction stage.

[0109] (2) The transformed element in the first column of the second row is compared in value with the transformed element in the second column of the second row. If the transformed element in the first column of the second row is greater than the transformed element in the second column of the second row in value, it is determined that the target object will not suffer from preeclampsia in the second preeclampsia prediction stage (such as 36 weeks of pregnancy). If the transformed element in the first column of the second row is less than the transformed element in the second column of the second row in value, it is determined that the target object will suffer from preeclampsia in the second preeclampsia prediction stage.

[0110] (3) Numerical comparison is performed between the transformed element in the first column of the third row and the transformed element in the second column of the third row, if the numerical value of the transformed element in the first column of the third row is greater than the numerical value of the transformed element in the second column of the third row, it is determined that the target object will not have eclampsia in the third eclampsia prediction stage (such as 38 weeks of pregnancy); if the numerical value of the transformed element in the first column of the third row is less than the numerical value of the transformed element in the second column of the third row, it is determined that the target object will have eclampsia in the third eclampsia prediction stage.

[0111] Thus, the eclampsia prediction of the target object is realized based on the fusion result of the inspection data of the target object.

[0112] On this basis, please refer to the following scenario embodiments:

[0113] Please refer to Figure 7 , Figure 7 The overall framework diagram of the inspection data fusion method provided by the embodiments of the present application is as follows:

[0114] 1. Data acquisition.

[0115] First, the basic data information of the pregnant woman herself and the relevant feature information of different pregnancy periods under different inspection modalities are integrated, and the output is the sum information of each inspection modality. Then, the sum information between multiple modalities is integrated by using a mask attention model (mask network + attention network), and then the eclampsia occurring in different pregnancy periods is predicted.

[0116] Among them, the basic data information of the pregnant woman can include but is not limited to: age, race, height, weight, smoking, type 1 diabetes / type 2 diabetes insulin treatment, mode of conception, chronic diabetes, systemic lupus erythematosus, antiphospholipid syndrome, preeclampsia family history, pregnancy history, etc.

[0117] Various inspection modalities can include but are not limited to:

[0118] Blood pressure and weight: Since the collection of blood pressure and weight data is relatively easy, the weight and the corresponding diastolic pressure and systolic pressure are collected within the data collection time of different inspection modalities.

[0119] Blood test information: including but not limited to hemoglobin, red blood cell count, white blood cell count, platelet count, mean corpuscular volume, mean corpuscular hemoglobin concentration, mean corpuscular hemoglobin content, while the detection indexes significantly related to eclampsia mainly include soluble tyrosine kinase 1, placental growth factor, soluble endothelin, placental protein 13, pregnancy-associated plasma protein a and vascular endothelial growth factor.

[0120] Kidney detection information: including but not limited to urine occult blood, urine white blood cells, urine ketone body, urine sugar, urine protein, and the detection indexes significantly related to preeclampsia mainly include urine β2-microglobulin and kidney injury molecule-1.

[0121] Liver function detection information: including but not limited to transaminase, bilirubin, albumin and total protein, and alkaline phosphatase.

[0122] Uterus-related indicators: including but not limited to fetal heart detection, fetal size (biparietal diameter, head circumference, abdominal circumference, femur length and fetal weight), uterine artery blood flow (uterine artery pressure pulsatility index, resistance index, systolic and diastolic flow rate ratio), etc.

[0123] Heart detection information: including but not limited to electrocardiogram (ECG) information. At the same time, the information highly related to preeclampsia is echocardiogram examination information, mainly including left atrial end-systolic diameter, left ventricular diastolic posterior wall thickness, interventricular septal diastolic end thickness, left ventricular diastolic end diameter, diastolic end reverse blood flow peak, relative wall thickness calculation, left ventricular mass index, cardiac index, left ventricular ejection fraction and cardiac output.

[0124] Pregnant women's ophthalmic artery Doppler ultrasound: including but not limited to systolic peak velocity, diastolic end velocity, resistance index, pulsatility index, ratio of second peak flow rate to initial peak flow rate.

[0125] 2. Multi-feature fusion processing based on time characteristics.

[0126] Please refer to Figure 8 , Figure 8 The flowchart of the multi-feature data fusion method provided by the embodiments of the present application can include the following specific implementation processes:

[0127] 1. Construct a two-dimensional gestational age representation system: decompose the gestational period into gestational weeks (1-41 weeks) and gestational days (0-6 days) two orthogonal characteristics, which not only retains the macro-pregnancy stage characteristics, but also realizes the gestational day level time resolution. Moreover, for the same type of examination features of different examination periods (gestational weeks + gestational days), the same type of examination features of different examination periods are fused by adopting a near-large and far-small weight setting method.

[0128] 2. For the common missing value problem of clinical data, an intelligent mask processing technology based on the Transformer architecture is adopted to process the missing data through the attention mask mechanism.

[0129] For the maternal electrocardiogram information and fetal heart monitoring of the uterus, a two-layer attention model is used to collect relevant features, and the feature information is compressed to the same size as other features. The time sequence feature information is spliced into other feature information. When the ultrasound image index is missing and there is image data, relevant features can be extracted by processing the ultrasound image data. The number and dimension of the index features are used to output the relevant feature matrix. The TimeSformer network is used to complete the extraction of the relevant features.

[0130] 3. Multimodal fusion processing.

[0131] Please refer to Figure 9 , Figure 9 The flowchart of the multimodal data fusion method provided by the embodiments of the present application can include: after completing the multi-feature fusion of each single examination modality, in the case of a known missing modality, using a mask attention model, integrating information under different examination modalities, and outputting the final integrated information. The multimodal fusion process can be to stack the data under multiple examination modalities into a matrix block, and then use a multi-head attention module to perform multimodal fusion on the matrix block.

[0132] 4. Overall model loss design.

[0133] First, the loss function can use binary cross-entropy loss. In order to improve the robustness of different examination modalities, after extracting the information of each examination modality, a fully connected layer is added and a sigmoid activation function is added, and the loss of each examination modality and the final variance is calculated, L=-y×log(y pred ) - (1-y) × log (1-y pred ); wherein L represents the loss of a certain examination modality and the final variance, y is the actual label (whether to have eclampsia in the corresponding gestational week), y pred is the prediction result.

[0134] Further, the primary loss is the sum of the losses of all examination modalities and the final variance, that is, the sum of K Ls, and K is the total number of the current examination modalities.

[0135] Finally, the final loss is the weighted sum of the primary loss and the secondary loss, that is, the final loss = 0.4 × primary loss + 0.6 × secondary loss.

[0136] It can be seen that the eclampsia prediction method based on the neural network model provided by the embodiments of the present application, on the basis of realizing effective integration of target object examination data based on missing data by using a neural network model, further realizes eclampsia prediction based on modality fusion results by using a prediction network in the neural network model, thereby effectively improving the accuracy of the eclampsia prediction result.

[0137] The embodiment of the present application provides a computer device.

[0138] Please refer to Figure 10 , Figure 10 The structural schematic diagram of the computer device provided by the present application can comprise:

[0139] The memory 11 is used for storing a computer program.

[0140] The processor 10 is used for implementing the steps of the data fusion method or the steps of the eclampsia prediction method when executing the computer program.

[0141] As shown in Figure 10 , the structural schematic diagram of the computer device can comprise a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 can complete mutual communication through the communication bus 13.

[0142] In the embodiment of the present application, the processor 10 can be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0143] The processor 10 can call the program stored in the memory 11, and specifically, the processor 10 can execute the operations in the embodiment of the data fusion method or the eclampsia prediction method.

[0144] The memory 11 is used for storing one or more programs, and the program can comprise program code including computer operation instructions. In the embodiment of the present application, the memory 11 at least stores a program for implementing the following functions:

[0145] Obtaining the inspection data of the target object; wherein the inspection data comprises multiple types of inspection modalities and multiple types of inspection features under each inspection modality; for at least part of the multiple types of inspection modalities, a first neural network submodel in the neural network model is used to firstly perform mask processing on the multiple types of inspection features under the inspection modality according to the feature missing result under the inspection modality, and then perform fusion processing on the multiple types of inspection features after the mask processing, to obtain a feature fusion result under the inspection modality; a second neural network submodel in the neural network model is used to firstly perform mask processing on the feature fusion result under the multiple types of inspection modalities according to the modality missing result, and then perform fusion processing on the feature fusion result after the mask processing, to obtain a modality fusion result of the target object;

[0146] Or,

[0147] Obtaining inspection data of a target object; wherein the inspection data comprises multiple types of inspection modalities and multiple types of inspection features under each inspection modality; for at least part of the multiple types of inspection modalities, a first neural network sub-model in the neural network model first performs mask processing on the multiple types of inspection features under the inspection modality according to a feature missing result under the inspection modality, and then performs fusion processing on the multiple types of inspection features after the mask processing to obtain a feature fusion result under the inspection modality; a second neural network sub-model in the neural network model first performs mask processing on the feature fusion results under the multiple types of inspection modalities according to a modality missing result, and then performs fusion processing on the feature fusion results after the mask processing to obtain a modality fusion result of the target object; and a prediction network in the neural network model processes the modality fusion result to obtain a prediction result of eclampsia of the target object.

[0148] In a possible implementation, the memory 11 can include a program storage area and a data storage area, where the program storage area can store an operating system and application programs required by at least one function, etc.; and the data storage area can store data created during use.

[0149] In addition, the memory 11 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device or other volatile solid-state storage device.

[0150] The communication interface 12 can be an interface of a communication module, configured to connect with other devices or systems.

[0151] Of course, it needs to be explained that, Figure 10 The structures shown do not constitute a limitation on the computer device in the embodiments of the present application, and in actual applications, the computer device can include more or fewer components than Figure 10 those shown, or combine certain components.

[0152] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the above inspection data fusion methods.

[0153] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0154] For the computer readable storage medium provided by the present application, refer to the above method embodiments, and the present application will not be repeated here.

[0155] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0156] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0158] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for fusion of inspection data based on a neural network model, characterized in that, include: Obtain inspection data of the target object; wherein, the inspection data includes multiple inspection modalities and multiple inspection features under each inspection modality; For at least some of the multi-type inspection modes, the first neural network sub-model in the neural network model first performs masking processing on the multi-type inspection features under the inspection mode based on the feature missing results under the inspection mode, and then performs fusion processing on the masked multi-type inspection features to obtain the feature fusion result under the inspection mode. The second neural network sub-model in the neural network model first performs masking processing on the feature fusion results under the multi-type inspection modality based on the modality missing results, and then performs fusion processing on the masked feature fusion results to obtain the modality fusion result of the target object.

2. The inspection data fusion method according to claim 1, characterized in that, When the multi-type inspection features under the inspection modality are the inspection features of the inspection modality in multiple different inspection cycles, after obtaining the inspection data of the target object, the method further includes: For at least some of the multi-type inspection modes, multiple inspection features of the same type in multiple different inspection cycles of the inspection mode are fused to obtain the fusion result of the corresponding type of inspection features, and this result is used as the multi-type inspection features under the inspection mode.

3. The inspection data fusion method according to claim 2, characterized in that, The inspection modality is fused across multiple inspection features of the same type in multiple different inspection cycles to obtain a fusion result of the corresponding type of inspection features, including: The weight values ​​of multiple inspection features of the same type within the multiple inspection cycles are determined according to the time sequence of the multiple inspection cycles; the weight values ​​increase from the earliest to the latest in the time sequence of the inspection cycles. Based on the weight values, multiple inspection features of the same type within multiple inspection cycles are weighted and summed to obtain the fusion result of the corresponding type of inspection features.

4. The inspection data fusion method according to claim 2, characterized in that, The examination data is the pregnancy examination data of the target subject; Accordingly, the examination cycle includes gestational weeks and gestational days.

5. The inspection data fusion method according to any one of claims 1 to 4, characterized in that, Both the first neural network sub-model and the second neural network sub-model include a mask network and an attention network.

6. The inspection data fusion method according to claim 5, characterized in that, Before masking the multi-type inspection features in the inspection modality based on the feature loss results in the inspection modality, the method further includes: Based on the preset inspection feature type and the multiple types of inspection features under the inspection mode, the missing inspection feature type under the inspection mode is determined, so as to take the missing inspection feature type under the inspection mode as the feature missing result under the inspection mode; Accordingly, based on the feature loss results under the inspection modality, the multi-type inspection features under the inspection modality are masked, including: Using the masking network, a feature missing vector for the inspection mode is first constructed based on the multi-type inspection features under the inspection mode and the feature missing result. Then, the multi-type inspection features under the inspection mode are masked using the feature missing vector to obtain the masked multi-type inspection features.

7. The inspection data fusion method according to claim 6, characterized in that, The process of fusing the multi-type inspection features after masking to obtain the feature fusion result under the inspection modality includes: Using the attention network, the first learning matrix is ​​first updated based on the missing feature vector, and then the updated first learning matrix is ​​used to fuse the multi-type inspection features after the masking process to obtain the feature fusion result under the inspection modality.

8. The inspection data fusion method according to claim 5, characterized in that, Before masking the feature fusion results under the multi-type inspection modality based on the modality missing results, the method further includes: The missing inspection modality type is determined based on the preset inspection modality type and the multiple inspection modal types, and the missing inspection modality type is used as the modality missing result; Accordingly, the feature fusion results under the multi-type inspection modalities are masked based on the modality missing results, including: Using the masking network, a mode missing vector is first constructed based on the multi-type inspection mode and the mode missing result. Then, the mode missing vector is used to mask the feature fusion result under the multi-type inspection mode to obtain the masked feature fusion result.

9. The inspection data fusion method according to claim 8, characterized in that, The process of fusing the feature fusion results after masking to obtain the modality fusion result of the target object includes: Using the attention network, the second learning matrix is ​​first updated according to the modality missing vector, and then the updated second learning matrix is ​​used to fuse the feature fusion result after the masking process to obtain the modality fusion result of the target object.

10. The inspection data fusion method according to claim 5, characterized in that, Both the first neural network sub-model and the second neural network sub-model further include a fully connected network.

11. The inspection data fusion method according to claim 10, characterized in that, After fusing the multi-type inspection features after masking to obtain the feature fusion result under the inspection modality, the process further includes: Using the fully connected network, the feature fusion result under the inspection modality is compressed according to the first feature size to obtain the feature compression result under the inspection modality.

12. The inspection data fusion method according to claim 10, characterized in that, After performing fusion processing on the feature fusion results after masking to obtain the modality fusion result of the target object, the method further includes: Using the fully connected network, the modal fusion result of the target object is compressed according to the second feature size to obtain the modal compression result of the target object.

13. A method for predicting eclampsia based on a neural network model, characterized in that, include: Obtain inspection data of the target object; wherein, the inspection data includes multiple inspection modalities and multiple inspection features under each inspection modality; For at least some of the multi-type inspection modes, the first neural network sub-model in the neural network model first performs masking processing on the multi-type inspection features under the inspection mode based on the feature missing results under the inspection mode, and then performs fusion processing on the masked multi-type inspection features to obtain the feature fusion result under the inspection mode. The second neural network sub-model in the neural network model first performs masking processing on the feature fusion results under the multi-type inspection modality based on the modality missing results, and then performs fusion processing on the masked feature fusion results to obtain the modality fusion result of the target object; The modality fusion result is processed by the prediction network in the neural network model to obtain the eclampsia prediction result for the target object.

14. A computer device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the examination data fusion method as claimed in any one of claims 1 to 12 or the steps of the eclampsia prediction method as claimed in claim 13 when executing the computer program.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the examination data fusion method as described in any one of claims 1 to 12 or the steps of the eclampsia prediction method as described in claim 13.

Images