A cross-modal early pregnancy fetal congenital heart disease diagnosis system and method

Abstract

The present application proposes a cross-modal early pregnancy fetal congenital heart disease diagnosis system and method, which relates to the field of medical technology. The system includes semantic segmentation, matrix construction, expert scoring, congenital heart disease classification, artificial intelligence and result fusion modules. The system identifies key fetal heart anatomical structures from images, generates a variety of matrix diagrams, combines expert knowledge and classification algorithms for prediction, and displays the results through an artificial intelligence interface. Medical staff can enter diagnosis and treatment opinions based on this, and the system integrates expert scores, prediction results and diagnosis and treatment decisions to generate the final prediction results. This system can segment and locate the key anatomical structures of the standard section of early pregnancy fetal congenital heart disease from the input image, and improve the accuracy and efficiency of early pregnancy fetal congenital heart disease diagnosis through multimodal information fusion, expert knowledge scoring and cross-modal diagnosis and treatment framework.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a cross-modal early pregnancy fetal congenital heart disease diagnosis system and method. Background Art

[0002] Accurate diagnosis of early pregnancy congenital heart disease is crucial to the health of the fetus. However, due to the small size of the fetus and the underdevelopment of some organs, traditional diagnosis is difficult, time-consuming and extremely dependent on the clinical experience of obstetricians and gynecologists. In other words, traditional diagnostic methods rely heavily on the professional knowledge and experience of doctors, and are highly subjective and inefficient. Therefore, there is an urgent need for a system that can automatically and accurately diagnose early pregnancy congenital heart disease to solve this problem. Summary of the invention

[0003] The purpose of the present application is to provide a cross-modal early pregnancy fetal congenital heart disease diagnosis system and method, which can segment and locate the key anatomical structures of the standard section of early pregnancy fetal congenital heart disease from the input image, and improve the accuracy and efficiency of early pregnancy fetal congenital heart disease diagnosis through multimodal information fusion, expert knowledge scoring and cross-modal diagnosis and treatment framework.

[0004] This application is implemented as follows:

[0005] In a first aspect, the present application provides a cross-modal early pregnancy fetal congenital heart disease diagnosis system, comprising:

[0006] A semantic segmentation module is used to segment and locate the key anatomical structures of the standard section of early pregnancy fetal congenital heart disease from the input image; the key anatomical structures include the four-chamber heart standard section and the three-vessel tracheal standard section. A matrix construction module is used to generate a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix, and a blood flow-key anatomical structure co-occurrence matrix diagram based on the key anatomical structures segmented and located. An expert knowledge scoring module is used to generate expert scoring results based on the preset expert knowledge rules using the key anatomical structures segmented and located and the blood flow classification matrix. A congenital heart disease classification module is used to generate classification prediction results based on the key anatomical structures segmented and located, the key anatomical structure co-occurrence matrix diagram, the blood flow classification matrix, and the blood flow-key anatomical structure co-occurrence matrix diagram. An artificial intelligence module is used to display the key anatomical structures segmented and located, the blood flow classification results, the expert scoring results, and the classification prediction results for medical staff to enter corresponding diagnosis and treatment opinions and diagnosis and treatment decisions; the blood flow classification results are the natural language representation of the blood flow classification matrix. The result fusion module is used to integrate the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff to generate the final prediction results.

[0007] Furthermore, based on the aforementioned scheme, the semantic segmentation module is further used to fuse the input two-dimensional grayscale image and color Doppler image of the same section, and segment the key anatomical structures required for standard section detection of fetal congenital heart disease in early pregnancy based on the image obtained by the fusion process.

[0008] Further, based on the aforementioned scheme, the matrix construction module includes: a category interaction diagram calculation unit, which is used to calculate the category interaction diagram of the four-chamber heart standard section and the three-vessel tracheal standard section according to the key anatomical structures located by segmentation. A dictionary matrix construction unit, which is used to construct a dictionary matrix according to the category values ​​of the key anatomical structures located by segmentation. A co-occurrence matrix diagram generation unit, which is used to generate the key anatomical structure co-occurrence matrix diagram of the four-chamber heart standard section and the three-vessel tracheal standard section according to the dictionary matrix and the category interaction diagram. A blood flow classification matrix generation unit, which is used to predict the blood flow quantity, shape and color of the four-chamber heart standard section and the three-vessel tracheal standard section according to the key anatomical structures located by segmentation, and generate the corresponding blood flow classification matrix. A blood flow-key anatomical structure co-occurrence matrix diagram generation unit, which is used to combine the blood flow classification matrix with the key anatomical result co-occurrence matrix diagram to generate a blood flow-key anatomical structure co-occurrence matrix diagram.

[0009] Furthermore, based on the aforementioned scheme, the neural network structure of the blood flow classification matrix generation unit includes multiple fully connected layers, and the prediction loss of blood flow quantity, shape and color is calculated through the Softmax cross entropy loss function.

[0010] Further, based on the aforementioned scheme, the expert knowledge rules are scoring rules constructed according to the evaluation results of key features in key anatomical structures, and the key features include whether there is asymmetry of color blood flow bundles in a single ventricle and left and right ventricles in the four-chamber heart standard section, whether only one vascular arch is displayed in the three-vessel trachea standard section, and whether there is abnormal U-shaped vascular arrangement.

[0011] Further, based on the aforementioned scheme, the artificial intelligence module is further used to adjust the selection of the input image in response to the standard section selection instructions of the medical staff.

[0012] Furthermore, based on the aforementioned scheme, the result fusion module combines the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff as training data, and uses the AdaBoost algorithm to pre-train a strong classifier model.

[0013] In a second aspect, the present application provides a method comprising the following steps:

[0014] The key anatomical structures of the standard section of early pregnancy fetal congenital heart disease are segmented and located from the input image; the key anatomical structures include the four-chamber heart standard section and the three-vessel tracheal standard section. According to the key anatomical structures segmented and located, a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram are generated. According to the preset expert knowledge rules, the key anatomical structures segmented and located and the blood flow classification matrix are used to generate expert scoring results. According to the key anatomical structures segmented and located, the key anatomical structure co-occurrence matrix diagram, the blood flow classification matrix and the blood flow-key anatomical structure co-occurrence matrix diagram, a classification prediction result is generated. The expert scoring results, the classification prediction results and the diagnosis and treatment decisions entered by the medical staff are integrated to generate the final prediction results; the diagnosis and treatment decisions entered by the medical staff are based on the key anatomical structures segmented and located, the blood flow classification results, the expert scoring results and the classification prediction results, combined with the personal clinical diagnosis and treatment experience. The blood flow classification result is a natural language representation of the blood flow classification matrix.

[0015] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, a method as described in any one of the second aspects is implemented.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method as described in any one of the above-mentioned second aspects.

[0017] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0018] This application proposes a cross-modal early pregnancy fetal congenital heart disease diagnosis system that can comprehensively utilize key anatomical structure information, blood flow information and expert knowledge rules to more accurately identify fetal congenital heart disease. In addition, the collaborative work between the modules makes the entire diagnostic process more automated and efficient, allowing medical staff to quickly obtain the various results generated by the system and perform further diagnosis or treatment as needed. In addition, the artificial intelligence module provides an intuitive visual interface and interactive functions, allowing medical staff to effectively interact with the system. Medical staff will be able to make more accurate diagnoses together based on the results provided by the system and their own clinical experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a structural block diagram of an embodiment of a cross-modal early pregnancy fetal congenital heart disease diagnosis system of the present application;

[0021] Figure 2 This is a schematic diagram of the framework of an embodiment of a cross-modality early pregnancy fetal congenital heart disease diagnosis system of the present application;

[0022] Figure 3 A flowchart of an embodiment of a method of the present application;

[0023] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0024] Icon: 201, processor; 202, memory; 203, communication interface. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Example 1

[0029] See also Figure 1-2The cross-modal early pregnancy fetal congenital heart disease diagnosis system includes a semantic segmentation module, a matrix construction module, an expert knowledge scoring module, a congenital heart disease classification module, an artificial intelligence module and a result fusion module; wherein the semantic segmentation module is used to segment and locate the key anatomical structures of the standard section of early pregnancy fetal congenital heart disease from the input image; the key anatomical structures include the four-chamber heart standard section and the three-vessel trachea standard section. The matrix construction module is used to generate a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram according to the key anatomical structures segmented and located. The expert knowledge scoring module is used to generate an expert scoring result according to the preset expert knowledge rules using the key anatomical structures segmented and located and the blood flow classification matrix. The congenital heart disease classification module is used to generate a classification prediction result according to the key anatomical structures segmented and located, the key anatomical structure co-occurrence matrix diagram, the blood flow classification matrix and the blood flow-key anatomical structure co-occurrence matrix diagram. The artificial intelligence module is used to display the key anatomical structures located by segmentation, blood flow classification results, expert scoring results and classification prediction results, so that medical staff can enter the corresponding diagnosis and treatment opinions and treatment decisions; the blood flow classification results are the natural language representation of the blood flow classification matrix. The result fusion module is used to fuse the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff to generate the final prediction results.

[0030] In the cross-modal early pregnancy fetal congenital heart disease diagnosis system, the semantic segmentation module is first used to perform semantic segmentation on the input fetal heart ultrasound image to extract key anatomical structure information. This information is then passed to the matrix construction module as the basis for subsequent analysis. Among them, the input fetal heart ultrasound image includes a two-dimensional grayscale image and a color Doppler image of the same section. The two-dimensional grayscale image shows the basic structure and morphology of the fetal heart, while the color Doppler image provides blood flow information, which is helpful for analyzing heart function and abnormalities; the extracted key anatomical structures include the four-chamber heart standard section and the three-vessel trachea standard section.

[0031] Then, after receiving the key anatomical structure information, the matrix construction module will use this information to construct the key anatomical structure co-occurrence matrix, blood flow classification matrix and blood flow-key anatomical structure co-occurrence matrix. These matrix diagrams not only reflect the spatial relationship between the key anatomical structures, but also contain rich blood flow information. These matrix diagrams are then passed to the expert knowledge scoring module and the congenital heart disease classification module. Next, the expert knowledge scoring module will use the segmented key anatomical structures and blood flow classification matrix for preliminary diagnostic scoring based on the preset expert knowledge rules, and its expert scoring results will then be passed to the artificial intelligence module and the result fusion module for use as reference information. After receiving the key anatomical structures, key anatomical structure co-occurrence matrix diagram, blood flow classification matrix and blood flow-key anatomical structure co-occurrence matrix diagram, the congenital heart disease classification module will use the machine learning algorithm for training to generate classification prediction results, which are also passed to the artificial intelligence module and the result fusion module. For the artificial intelligence module, it is responsible for visually displaying the key anatomical structures, blood flow classification results, expert scoring results and classification prediction results identified by segmentation, and providing an interactive interface for medical staff to enter diagnosis and treatment opinions and decisions, which are then passed to the result fusion module.

[0032] Finally, after receiving the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff, the result fusion module will generate the final prediction result based on certain weight allocation and logical operations. This prediction result comprehensively considers multiple aspects of information and has higher accuracy and reliability.

[0033] That is to say, through precise data interaction between modules, the system can make comprehensive use of key anatomical structure information, blood flow information and expert knowledge rules to more accurately identify fetal congenital heart disease. Moreover, the collaborative work between modules makes the entire diagnostic process more automated and efficient, allowing medical staff to quickly obtain various results generated by the system and conduct further diagnosis or treatment as needed. In addition, the artificial intelligence module provides an intuitive visual interface and interactive functions, allowing medical staff to effectively interact with the system. Medical staff will be able to make more accurate diagnoses together based on the results provided by the system and their own clinical experience.

[0034] In summary, in view of the problem that the early pregnancy fetus is small and some organs are underdeveloped, it is difficult to effectively diagnose and treat early pregnancy congenital heart disease. In the above embodiment, the key anatomical structures are first segmented on the standard sections of the four-chamber heart section and the three-vessel tracheal section, and then a multi-scale co-occurrence matrix diagram is constructed on the standard sections of the four-chamber heart section and the three-vessel tracheal section respectively. Then, based on this, a fetal congenital heart disease classification model is constructed, which can make full use of color Doppler images to improve the diagnosis of early pregnancy fetal congenital heart disease. Next, expert knowledge is introduced to construct a knowledge scoring system for fetal congenital heart disease, which interacts with the fetal congenital heart disease classification model to output the diagnosis effect of fetal congenital heart disease. At the same time, by integrating the diagnosis and treatment opinions and decisions of medical staff (usually junior doctors), the diagnostic efficiency of the early pregnancy fetal congenital heart disease diagnosis and treatment model is further improved, so that the early pregnancy fetus can be screened by an artificial intelligence model alone, and medical staff can also be assisted in making more accurate diagnoses.

[0035] In one implementation of the present application, the semantic segmentation module is further used to fuse the input two-dimensional grayscale image and color Doppler image of the same section, and segment the key anatomical structures required for standard section detection of congenital heart disease in the early pregnancy fetus based on the image obtained by the fusion process.

[0036] In the above implementation, the two-dimensional grayscale image and the color Doppler image each provide different information. The two-dimensional grayscale image mainly reflects the anatomical structure of the fetal heart and its surrounding tissues, while the color Doppler image can show the direction and speed of blood flow. By fusing these two images, we can obtain a comprehensive image that contains both anatomical structure information and blood flow information. The semantic segmentation module will first preprocess the input two-dimensional grayscale image and color Doppler image, including image registration, denoising, enhancement and other steps to ensure the consistency of the two images in spatial position and improve their clarity and contrast. Then, the module will use an appropriate fusion algorithm (such as weighted average method, maximum method, Laplace pyramid fusion method, etc.) to fuse the two images into a new comprehensive image.

[0037] It should be noted that in order to fuse the two images, in the actual processing process, the two-dimensional grayscale image and color Doppler image of the same section can be spliced ​​left to right, then right to left, and finally the two spliced ​​images are stacked as the input of the neural network, so that the convolutional neural network can automatically learn the anatomical organ information of the two-dimensional grayscale image and the functional blood flow information of the color Doppler image, and thus output the fused image.

[0038] Then, after obtaining the fused composite image, the semantic segmentation module will use deep learning technology (such as convolutional neural network CNN) to segment it. By training a pre-designed neural network model, the model can automatically identify key anatomical structures in the composite image, such as the related structures of the four-chamber heart standard section and the three-vessel trachea standard section. During the segmentation process, the module will use the pixel-level information in the image to distinguish the key anatomical structures from the background or other non-critical structures through pixel-by-pixel classification. At the same time, because the fused image contains rich blood flow information, the module can also more accurately identify key anatomical structures related to blood flow.

[0039] By fusing two-dimensional grayscale images and color Doppler images and combining deep learning technology to segment key anatomical structures, the semantic segmentation module can more comprehensively obtain the anatomical information and blood flow information of the fetal heart and its surrounding tissues, thereby improving the accuracy of subsequent diagnosis. In addition, since the fused images contain rich anatomical and blood flow information, medical staff can perform more comprehensive diagnostic analysis based on this information. For example, they can observe the direction and speed of blood flow to assess whether the heart function is normal; they can also compare anatomical structures under different sections to detect potential abnormal changes. These comprehensive diagnostic analyses can provide medical staff with more accurate diagnostic basis and more complete treatment plans.

[0040] In one implementation of the present application, the matrix construction module includes a category interaction diagram calculation unit, a dictionary matrix construction unit, a co-occurrence matrix diagram generation unit, a blood flow classification matrix generation unit, and a blood flow-key anatomical structure co-occurrence matrix diagram generation unit. Among them, the category interaction diagram calculation unit is used to calculate the category interaction diagram of the four-chamber heart standard section and the three-vessel tracheal standard section according to the key anatomical structures located by segmentation. The dictionary matrix construction unit is used to construct a dictionary matrix according to the category values ​​of the key anatomical structures located by segmentation. The co-occurrence matrix diagram generation unit is used to generate the key anatomical structure co-occurrence matrix diagram of the four-chamber heart standard section and the three-vessel tracheal standard section according to the dictionary matrix and the category interaction diagram. The blood flow classification matrix generation unit is used to predict the blood flow quantity, shape and color of the four-chamber heart standard section and the three-vessel tracheal standard section according to the key anatomical structures located by segmentation, and generate the corresponding blood flow classification matrix. The blood flow-key anatomical structure co-occurrence matrix diagram generation unit is used to combine the blood flow classification matrix with the key anatomical result co-occurrence matrix diagram to generate the blood flow-key anatomical structure co-occurrence matrix diagram.

[0041] In the above implementation, the matrix construction module is composed of the following five core sub-units: a category interaction map calculation unit, a dictionary matrix construction unit, a co-occurrence matrix map generation unit, a blood flow classification matrix generation unit, and a blood flow-key anatomical structure co-occurrence matrix map generation unit. Among them, the category interaction map calculation unit is responsible for calculating the category interaction map of the four-chamber heart standard section and the three-vessel trachea standard section according to the key anatomical structures (such as ventricles, atria, blood vessels, etc.) located by segmentation. The category interaction map can intuitively display the spatial relationship and interaction between different anatomical structures. The dictionary matrix construction unit is responsible for constructing a dictionary matrix according to the category values ​​of the key anatomical structures located by segmentation. This matrix is ​​actually a collection of category indexes for subsequent rapid retrieval and matching of attribute information of related anatomical structures. The co-occurrence matrix map generation unit is responsible for generating the key anatomical structure co-occurrence matrix map of the four-chamber heart standard section and the three-vessel trachea standard section according to the dictionary matrix and the category interaction map. The co-occurrence matrix map can reflect the co-occurrence frequency and relative position of the anatomical structure in the image, which helps to identify potential structural abnormalities. The blood flow classification matrix generation unit is a prediction unit based on a deep learning or machine learning model. Based on the key anatomical structures segmented and located, it predicts the blood flow quantity, shape and color of the four-chamber heart standard section and the three-vessel trachea standard section, and generates the corresponding blood flow classification matrix. This matrix provides medical staff with detailed information about fetal cardiac hemodynamics. Finally, the blood flow-key anatomical structure co-occurrence matrix generation unit is responsible for combining the blood flow classification matrix with the key anatomical structure co-occurrence matrix to generate a comprehensive blood flow-key anatomical structure co-occurrence matrix. This diagram not only shows the co-occurrence of cardiac structures, but also incorporates blood flow information, providing medical staff with a more comprehensive and in-depth fetal heart assessment tool.

[0042] In one implementation of the present application, the neural network structure of the blood flow classification matrix generation unit includes multiple fully connected layers, and calculates the prediction loss of blood flow quantity, shape and color through the Softmax cross entropy loss function.

[0043] In the above implementation, the fully connected layer is used to capture the complex features of blood flow in the image and convert it into a vector with classification significance. These fully connected layers are usually stacked in a hierarchical structure from shallow to deep, from abstract to concrete. The initial fully connected layer is responsible for extracting low-level features in the image, such as edges, corners, etc.; as the layers go deeper, the network is gradually able to capture higher-level and more abstract features, such as the amount, shape, and color of blood flow. The last layer of the neural network is the output layer, which is responsible for converting the features learned by the network into specific classification results. In this application, the number of neurons in the output layer is equal to the number of blood flow categories that need to be classified, and the output value of each neuron represents the probability of the corresponding category.

[0044] In addition, in order to measure the accuracy of the neural network prediction results and adjust the network parameters accordingly, the present application adopts the Softmax cross entropy loss function. This function first converts the output of the neural network into a probability distribution (through the Softmax function), and then calculates the cross entropy loss between this probability distribution and the true label. The smaller the cross entropy loss, the closer the network's prediction results are to the actual situation. In this application, since it is necessary not only to classify the types of blood flow, but also to predict the attributes such as the quantity, shape and color of blood flow. Therefore, it is necessary to consider the prediction losses of these attributes in the loss function at the same time. Specifically, a separate neuron can be set for each attribute in the output layer, and the cross entropy loss between them and the true value can be calculated; then, these losses are weighted and summed to obtain the final loss function.

[0045] In summary, by using multiple fully connected layers to construct a neural network structure and combining it with the Softmax cross entropy loss function for training, the characteristics of blood flow in the image can be more accurately captured, classified and described, which helps to improve the accuracy and reliability of fetal congenital heart disease diagnosis. In addition, the neural network structure in this application can not only classify the types of blood flow, but also predict the properties of blood flow such as quantity, shape and color, which provides medical staff with richer diagnostic information and helps them understand the condition of the fetal heart more comprehensively.

[0046] In one implementation of the present application, the expert knowledge rules are scoring rules constructed based on the evaluation results of key features in key anatomical structures, wherein the key features include whether there is asymmetry of color blood flow bundles in a single ventricle and between left and right ventricles in the four-chamber heart standard section, whether only one vascular arch is displayed in the three-vessel trachea standard section, and whether there is abnormal U-shaped vascular arrangement.

[0047] When analyzing the key features of the four-chamber heart standard section, it is necessary to evaluate the single ventricle and the symmetry of the left and right ventricle color flow beams, that is, when observing the four-chamber heart standard section, pay attention to whether only one ventricular structure is displayed, which may indicate abnormal heart development, and compare the brightness, direction and distribution of the left and right ventricular color flow beams to evaluate whether there is asymmetry, which may reflect abnormal ventricular function or hemodynamics; and when analyzing the key features of the three-vessel trachea standard section, it is necessary to evaluate the number of vascular arches and the abnormal arrangement of U-shaped vessels, that is, when observing the three-vessel trachea standard section, pay attention to whether only one vascular arch is displayed, which is usually related to certain heart and great blood vessel malformations, and evaluate the arrangement relationship between the vascular arches and the trachea, and pay attention to whether there is abnormal U-shaped vascular arrangement, which may also be a sign of abnormal heart or vascular development.

[0048] Based on the evaluation results of the above key features, a scoring rule is constructed, and each key feature is assigned a different score according to its degree of abnormality. The higher the score, the more serious the abnormality. Then, by accumulating the scores of each key feature, a total score can be obtained to comprehensively evaluate the overall health of the fetal heart. For example, for the standard section of the four-chamber heart section, if there is a single ventricle, 1 point is scored, and if the color blood flow bundles of the left and right ventricles of the four-chamber heart are asymmetric, 2 points are scored. For the standard section of the three-vessel tracheal section, if the blood flow only shows one vascular arch, 1 point is scored, and if the blood vessels have abnormal U-shaped vascular arrangement, 2 points are scored. The total score of the two sections is 6 points. If it is not less than 4 points, it can be considered that the fetus has a higher probability of abnormal congenital heart disease. If it does not exceed 1 point, it is considered that the fetus has a higher probability of diagnosis without abnormality. Between 1 and 4 points, it is difficult to distinguish the diagnosis and treatment of fetal congenital heart disease.

[0049] In one implementation of the present application, the artificial intelligence module is further used to respond to the standard section selection instructions of the medical staff to adjust the selection of the input image to ensure the accuracy and efficiency of the subsequent analysis. In the diagnosis of fetal congenital heart disease, medical staff usually select specific ultrasound image sections for analysis based on clinical experience and diagnostic needs. These standard sections usually include four-chamber heart standard sections, three-vessel trachea standard sections, etc., which can clearly show the structure and blood flow of the fetal heart and its surrounding blood vessels. In order to support these operations of medical staff, a response function for standard section selection instructions can be provided in the artificial intelligence module. Medical staff can send standard section selection instructions to the artificial intelligence module through a user interface or voice commands.

[0050] Exemplarily, junior doctors can evaluate whether the standard sections of the four-chamber heart section and the three-vessel tracheal section selected by the congenital heart disease diagnosis and treatment system are accurate according to clinical knowledge. If correct, the standard sections selected by the early pregnancy fetal standard section detection model are used to continue evaluating whether the tested fetus has congenital heart disease. If incorrect, the standard sections of the four-chamber heart section and the three-vessel tracheal section are selected from the sections with the highest expert knowledge scores and deep learning classification model scores until an image is selected in which the current section is evaluated as a standard section by the congenital heart disease classification expert knowledge score, the deep learning classification model, and junior doctors.

[0051] In one implementation of the present application, the result fusion module combines the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff into training data, and uses the AdaBoost algorithm to pre-train a strong classifier model.

[0052] AdaBoost (Adaptive Boosting) is an iterative algorithm that improves the performance of weak classifiers by adjusting sample weights and combines multiple weak classifiers to form a strong classifier. In the above implementation, a strong classifier model is pre-trained by using the AdaBoost algorithm to train the training data set generated by the result fusion module. During the training process, the AdaBoost algorithm will iterate repeatedly, and each iteration will adjust the weight of the sample according to the performance of the current weak classifier. Specifically, for samples that are misclassified, the algorithm will increase their weights so that subsequent weak classifiers pay more attention to these difficult-to-classify samples. At the same time, the algorithm will also assign different weights to the weak classifiers according to their performance. The better the performance of the weak classifier, the greater the weight in the final strong classifier. After multiple iterations, the AdaBoost algorithm will generate a strong classifier model composed of multiple weak classifiers. This model can make full use of various information from experts, artificial intelligence modules and medical staff to perform more accurate and comprehensive analysis of ultrasound images, providing a more reliable basis for the diagnosis of fetal congenital heart disease and other diseases.

[0053] In order to enable those skilled in the art to understand the present application more intuitively, a specific example will be used here to illustrate.

[0054] like Figure 2 As shown, the present application first uses a semantic segmentation module to segment and locate the key anatomical structures required for the standard section detection of early pregnancy fetal congenital heart disease from the input image (grayscale image and Doppler image of the same section), including the left atrium, right atrium, left ventricle, right ventricle, mitral valve blood flow, tricuspid valve blood flow, aortic arch and arterial duct of the four-chamber heart standard section and the three-vessel trachea standard section. Exemplarily, the data can be preprocessed first, and the grayscale image and Doppler image of the four-chamber heart standard section or the three-vessel trachea standard section are fused as the input of the network. Then, the two-dimensional grayscale image and the color Doppler image of the same section are first spliced ​​left to right, and then spliced ​​right to left, and finally the two spliced ​​images are stacked as the input of the neural network, so that the convolutional neural network automatically learns the anatomical organ information of the two-dimensional grayscale image and the functional blood flow information of the color Doppler image.

[0055] Then, Deeplab v3 is used as the segmentation network to achieve the key anatomical structure segmentation of the four-chamber heart standard section and the three-vessel trachea standard section. In the encoder network part, ResNet50 is used as the backbone network of the encoder network part. The output is the last layer feature of the encoder module of the four-chamber heart standard section segmentation network. The last layer of features in the decoder module The last layer features of the encoder module of the three-vessel trachea standard section segmentation network The last layer of features in the decoder module Through the Argmax operation, the last layer feature of the decoder of the four-chamber heart standard section segmentation network can be And the last layer features of the decoder of the three-vessel trachea standard section segmentation network The key anatomical results segmentation prediction results P transformed into the four-chamber heart standard section and the three-vessel trachea standard section respectively s1 and P s2 .

[0056] Next, after obtaining the four-chamber heart standard section segmentation result P s1 And the three-vessel trachea standard section segmentation result P s2 After that, the category interaction diagram M of the four-chamber standard section is calculated respectively. in1 Interactive diagram of the categories of the three-vessel tracheal standard section M in2 .

[0057] For the construction of the dictionary matrix: First, construct an 8×8 dictionary matrix D. The category values ​​corresponding to the background, left atrium, right atrium, left ventricle, right ventricle, mitral valve blood flow, tricuspid valve blood flow, aortic arch and ductus arteriosus in the key anatomical structure segmentation prediction image are 0, 1, 2, 3, 4, 5, 6, 7, and 8 respectively. Let d kl ∈D(0<k≤8,0<l≤8), then the value d in the dictionary matrix kl , which can be defined as: d kl =10×k×l+k+l, that is, the value of the dictionary matrix D is obtained by interacting with the category values ​​of the key anatomical structures of congenital heart disease.

[0058] Among them, for different d in the dictionary matrix D kl Value, respectively traverse and search for the category interaction diagram M of the four-chamber heart standard section in1 Interactive diagram of the categories of the three-vessel tracheal standard section M in2 , we can get each d kl Value in M in1 and M in2 The number of times c appears in kl1 and c kl2 Then all c kl1 and c kl2 Replace the values ​​into d kl By looking at the corresponding positions in the figure, we can get two 8×8 four-chamber heart standard section key anatomical structure co-occurrence matrices C 11 Symbiosis matrix of key anatomical structures of the three-vessel tracheal standard section Figure C 12 .

[0059] In the process of obtaining the blood flow classification matrix: use the color Doppler ultrasound images of the four-chamber heart standard section and the three-vessel trachea standard section to train two 1×3 four-chamber heart standard section blood flow classification matrices M b1 and three-vessel tracheal standard section blood flow classification matrix M b2 .

[0060] For example, the encoder feature behind the four-chamber standard view Two fully connected layers are connected in series, with the number of output channels being 2048 and 512, to reduce the dimension of features. At the same time, a ReLU activation function is added after these two fully connected layers to improve the nonlinear ability of the network. Then, three more fully connected layers are connected, with output channels being 3D, 3D, and 2D, to predict blood flow quantity, blood flow shape, and blood flow color, respectively.

[0061] During training, the three fully connected layers used to predict blood flow quantity, blood flow shape, and blood flow color are calculated through the Softmax cross entropy loss function and the corresponding annotation information. The loss functions are denoted as L bn1 , L bs1 and L bc1 The blood flow loss classification of the four-chamber standard section can be defined as: L b1 =L bn1 +L bs1 +L bc1 Then, the three fully connected layers are subjected to argmax operation to obtain the three predicted values ​​p of blood flow quantity, blood flow shape and blood flow color. bn1 、p bs1 and p bc2 Finally, the three predicted values ​​of the blood flow classification network are combined into a 1×3 four-chamber heart standard section blood flow classification matrix M b1 Similarly, the encoder features of the three-vessel tracheal standard section The blood flow classification matrix M of the three-vessel tracheal standard section can be calculated b2 Blood flow classification loss L of the three-vessel tracheal standard section b2 .

[0062] In the process of obtaining the blood flow-key anatomical structure co-occurrence matrix: take C 11 and C 12 The 36 elements in the upper right corner form a 36×1 matrix C 21 and C 22 Then let C 21 and C 22 The values ​​greater than 0 in the matrix are equal to 1, and we can get a 36×1 0,1 matrix C 31 and C 32 Finally, the 36×1 matrix C 31and the 1×3 matrix M b1 Multiplying them together can get the 36×3 blood flow-four-chamber heart standard section key anatomical structure co-occurrence matrix C 41 , the 36×1 matrix C 32 and the 1×3 matrix M b2 Multiplying them together can get the 36×3 blood flow-three-vessel tracheal standard section key anatomical structure co-occurrence matrix C 42 .

[0063] Without the intervention of junior doctors, two sets of analysis results, expert knowledge scoring and deep learning classification, can be obtained based on the results of multi-angle standard section key anatomical structure segmentation and multi-scale formula matrix diagram to determine whether the fetus has congenital heart disease.

[0064] Among them, when using the expert knowledge scoring module for expert scoring: for the standard section of the four-chamber heart section, if there is a single ventricle, 1 point is scored, and if the color blood flow bundles of the left and right ventricles of the four-chamber heart are asymmetric, 2 points are scored. For the standard section of the three-vessel tracheal section, if the blood flow only shows one vascular arch, 1 point is scored, and if the blood vessels have abnormal U-shaped vascular arrangement, 2 points are scored. The total score of the two sections is 6 points. If it is not less than 4 points, it can be considered that the fetus has a higher probability of congenital heart disease abnormalities. If it does not exceed 1 point, it is considered that the fetus is more likely to be diagnosed with no abnormalities. Between 1 and 4 points, it is difficult to judge the diagnosis and treatment of fetal congenital heart disease. Among them, whether the standard section of the four-chamber heart section has a single ventricle can be segmented by the four-chamber heart standard section. The result P s1 The symmetry of the color blood flow beams of the left and right ventricles of the four-chamber heart can be obtained from the blood flow classification matrix M of the standard section of the four-chamber heart. b1 The blood flow quantity and shape of the three-vessel tracheal standard section can be obtained from the blood flow classification matrix M of the three-vessel tracheal standard section. b2 Obtained in.

[0065] When using the congenital heart disease classification module to classify congenital heart disease: the early pregnancy fetal congenital heart disease classification module uses the four-chamber heart standard section segmentation result P s1 , three-vessel trachea standard section segmentation result P s2 、 Symbiosis matrix of key anatomical structures of standard sections of four-chamber heart C 11 , three-vessel tracheal standard section key anatomical structure co-occurrence matrix diagram C 12 , blood flow-four-chamber heart standard section key anatomical structure symbiosis matrix C 41 and blood flow-three-vessel tracheal standard section key anatomical structure co-occurrence matrix C 42 As input to predict whether the fetus has congenital heart disease. The specific process is as follows:

[0066] 1) The four-chamber standard section segmentation result P s1 And the three-vessel trachea standard section segmentation result P s2Cascade, followed by a fully connected layer with a channel number of 512 dimensions, to obtain a 512-dimensional feature vector F sh1 .

[0067] 2) The 8×8 four-chamber heart standard section key anatomical structure co-occurrence matrix C 11 Symbiosis matrix of key anatomical structures of the three-vessel tracheal standard section Figure C 12 And 36×3 blood flow-four-chamber heart standard section key anatomical structure symbiosis matrix C 41 and blood flow-three-vessel tracheal standard section key anatomical structure co-occurrence matrix C 42 They are converted into feature vectors of 64, 64, 108 and 108 dimensions respectively and concatenated to obtain a 344-dimensional feature vector F sh2 .

[0068] 3) The 512-dimensional feature vector F sh1 and the 344-dimensional feature vector F sh2 , followed by a ReLu activation function, outputting a 856-dimensional feature vector F sh3 .

[0069] 4) In the 856-dimensional feature vector F sh3 Followed by a fully connected layer with 256 output channels and a ReLu activation function, we get a 256-dimensional feature vector F sh4 .

[0070] 5) In the 256-dimensional feature vector F sh4 A fully connected layer with an output channel of 1 is then connected to obtain a 1-dimensional feature. Then, a Softmax operation is performed on this 1-dimensional feature to normalize its value to the interval [0,1], and the prediction value p for fetal congenital heart disease diagnosis is obtained. ch , used to determine whether the fetus being tested is sick.

[0071] 6) Finally, calculate the loss of the congenital heart disease classification task. Unlike the problem of large differences between positive and negative samples in the standard section classification task, the number of positive and negative samples for congenital heart disease diagnosis can be set manually. Therefore, the application uses L2 loss to reduce the prediction value p of fetal congenital heart disease diagnosis to ch The loss is calculated with the corresponding annotation information.

[0072] Then, in the test phase, two maximum thresholds and minimum thresholds are set to predict whether the current fetus has congenital heart disease. The maximum threshold is set to 0.7 and the minimum threshold is set to 0.3. When the predicted value is greater than 0.7, it is considered that the tested fetus has a high probability of having congenital heart disease. If the predicted value is less than 0.3, it is considered that the tested fetus has a high probability of being normal. If the threshold is between [0.3, 0.7], the module cannot determine whether the tested fetus has congenital heart disease.

[0073] In clinical applications, only when both the knowledge scoring and the congenital heart disease classification module detect congenital heart disease, do we determine that the tested fetus has a higher probability of having congenital heart disease. If the test results of the expert knowledge scoring and the congenital heart disease classification module both show that the fetus is normal, then the fetus is determined not to have congenital heart disease. In other cases, the system will show that it is impossible to determine whether the fetus has congenital heart disease, and a doctor is required to make further diagnosis. That is, there are three situations that require further diagnosis: 1) One of the test results of the expert knowledge scoring and the congenital heart disease classification module shows that the fetus has congenital heart disease or is normal, and the other shows that it cannot be determined; 2) Both the expert knowledge scoring and the congenital heart disease classification module show that they cannot be determined; 3) One of the expert knowledge scoring and the congenital heart disease classification module shows normal, and the other shows that the fetus has congenital heart disease.

[0074] In order to further improve the accuracy of diagnosis and treatment of fetal congenital heart disease in early pregnancy, the present application provides a cross-modal early pregnancy fetal congenital heart disease diagnosis and treatment system with collaborative constraints of junior doctors, which integrates the diagnosis results of junior doctors to more accurately evaluate the diagnostic efficacy of early pregnancy fetal congenital heart disease.

[0075] 1) First, junior doctors, with the assistance of the early pregnancy fetal standard section detection module, evaluate whether the standard sections of the four-chamber heart section and the three-vessel tracheal section selected by the congenital heart disease diagnosis and treatment system are accurate according to clinical knowledge. If correct, continue to evaluate whether the tested fetus has congenital heart disease according to the standard sections selected by the early pregnancy fetal standard section detection module. If incorrect, continue to select the standard sections of the four-chamber heart section and the three-vessel tracheal section from the sections with high expert knowledge scores and deep learning classification module scores until the images are selected for which the current section is evaluated as the standard section by the congenital heart disease classification expert knowledge score, the deep learning classification module and the junior doctors.

[0076] 2) The standard sections of the four-chamber heart section and the three-vessel tracheal section screened by junior doctors are sent to the early pregnancy fetal congenital heart disease classification module, and the key anatomical structure segmentation results and blood flow classification results of the four-chamber heart standard section and the three-vessel tracheal standard section, the congenital heart disease classification expert scoring results and the congenital heart disease classification module classification results are output.

[0077] 3) Then, junior doctors are asked to use the key anatomical structure segmentation results and blood flow classification results of the four-chamber heart standard section and the three-vessel trachea standard section output by the early pregnancy fetal congenital heart disease classification module, as well as the congenital heart disease classification expert scoring results and the congenital heart disease classification module classification results as priors, and combine their personal clinical diagnosis and treatment experience to output in a standardized format whether the heart axis of the four-chamber heart standard section image is abnormal, whether the ventricle is single, whether the blood flow is symmetrical, whether the ventricular septum is defective, and the blood flow quantity and shape of the three-vessel trachea standard section image. The doctors then make the final diagnosis and treatment decision, that is, whether the patient is sick.

[0078] 4) For the decisions made by junior doctors with the assistance of the AI ​​system, they are combined with the expert knowledge score and the output results of the congenital heart disease classification module, and a strong classifier module is trained through the AdaBoost algorithm to output whether the fetus has congenital heart disease.

[0079] 5) Then, a cross-modal congenital heart disease classification module was constructed. For the junior doctors’ opinions on whether the heart axis of the four-chamber heart standard section is abnormal, whether the ventricle is single, whether the blood flow is symmetrical, whether the ventricular septum is defective, and the blood flow quantity and shape of the three-vessel tracheal standard section, the Text2Vec module was first used to convert the junior doctors’ textual opinions on the fetal four-chamber heart standard section and the three-vessel tracheal standard section into two one-dimensional feature vectors P jt1 and P jt2 .

[0080] 6) The feature vector P jt1 and P jt2 Cascade, followed by a fully connected layer with an output channel of 512 and a ReLu activation function, to obtain a 512-dimensional feature vector P jt3 .

[0081] 7) Segmentation results using the four-chamber standard section P s1 , three-vessel trachea standard section segmentation result P s2 , Four-chamber heart standard section blood flow classification matrix M b1 and blood flow classification matrix M of the three-vessel tracheal standard section b2 As input, output is a 778-dimensional feature vector F sh3 For the specific process, please refer to the aforementioned congenital heart disease classification section.

[0082] 8) The 512-dimensional feature vector P jt3 and the 778-dimensional feature vector F sh3 Cascade, followed by a fully connected layer with 256 output channels and a ReLu activation function, to obtain a 256-dimensional feature vector P jt4 .

[0083] 9) In the 256-dimensional feature vector P jt4 Followed by a fully connected layer with an output channel of 1 and a Softmax layer, a 1-dimensional feature P in the interval [0,1] is obtained. jt5 , used to determine whether the fetus being tested is sick.

[0084] 10) Then, P is calculated by L2 loss jt5The corresponding label information is lost. In the test phase, the module is judged whether to detect congenital heart disease by setting a threshold. When the predicted value is greater than 0.7, it is considered that the tested fetus has a high probability of having congenital heart disease. If the predicted value is less than 0.3, it is considered that the tested fetus has a high probability of being normal. If the threshold is between [0.3, 0.7], the module cannot determine whether the tested fetus has congenital heart disease.

[0085] 11) Finally, in clinical applications, the fetal congenital heart disease diagnosis and treatment system will output two results, one is the result of the AdaBoost strong classifier module output, and the other is the cross-modal congenital heart disease classification module. Only when the output results of the two modules are consistent can it be determined whether the fetus has congenital heart disease. Otherwise, it is recommended that junior doctors change the standard section for re-testing, or ask senior doctors to conduct further examinations.

[0086] Example 2

[0087] See also Figure 3 , the present application embodiment provides a method, which includes the following steps:

[0088] The key anatomical structures of the standard section of early pregnancy fetal congenital heart disease are segmented and located from the input image; the key anatomical structures include the four-chamber heart standard section and the three-vessel tracheal standard section. According to the key anatomical structures segmented and located, a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram are generated. According to the preset expert knowledge rules, the key anatomical structures segmented and located and the blood flow classification matrix are used to generate expert scoring results. According to the key anatomical structures segmented and located, the key anatomical structure co-occurrence matrix diagram, the blood flow classification matrix and the blood flow-key anatomical structure co-occurrence matrix diagram, a classification prediction result is generated. The expert scoring results, the classification prediction results and the diagnosis and treatment decisions entered by the medical staff are integrated to generate the final prediction results; the diagnosis and treatment decisions entered by the medical staff are based on the key anatomical structures segmented and located, the blood flow classification results, the expert scoring results and the classification prediction results, combined with the personal clinical diagnosis and treatment experience. The blood flow classification result is a natural language representation of the blood flow classification matrix.

[0089] For the specific implementation process of the above method, please refer to the cross-modal early pregnancy fetal congenital heart disease diagnosis system provided in Example 1, which will not be repeated here.

[0090] Example 3

[0091] See also Figure 4, the embodiment of the present application provides an electronic device, which includes at least one processor 201 and at least one memory 202; wherein the processor 201 and the memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction; the memory 202 stores program instructions that can be executed by the processor 201, and the processor 201 calls the program instructions to execute a method. For example, to implement:

[0092] The key anatomical structures of the standard section of early pregnancy fetal congenital heart disease are segmented and located from the input image; the key anatomical structures include the four-chamber heart standard section and the three-vessel tracheal standard section. According to the key anatomical structures segmented and located, a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram are generated. According to the preset expert knowledge rules, the key anatomical structures segmented and located and the blood flow classification matrix are used to generate expert scoring results. According to the key anatomical structures segmented and located, the key anatomical structure co-occurrence matrix diagram, the blood flow classification matrix and the blood flow-key anatomical structure co-occurrence matrix diagram, a classification prediction result is generated. The expert scoring results, the classification prediction results and the diagnosis and treatment decisions entered by the medical staff are integrated to generate the final prediction results; the diagnosis and treatment decisions entered by the medical staff are based on the key anatomical structures segmented and located, the blood flow classification results, the expert scoring results and the classification prediction results, combined with the personal clinical diagnosis and treatment experience. The blood flow classification result is a natural language representation of the blood flow classification matrix. Among them, the memory 202 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0093] The processor 201 may be an integrated circuit chip with signal processing capability. The processor 201 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0094] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0095] Example 4

[0096] The present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by the processor 201, a method is implemented. For example, the following is implemented:

[0097] The key anatomical structures of the standard section of early pregnancy fetal congenital heart disease are segmented and located from the input image; the key anatomical structures include the four-chamber heart standard section and the three-vessel tracheal standard section. According to the key anatomical structures segmented and located, a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram are generated. According to the preset expert knowledge rules, the key anatomical structures segmented and located and the blood flow classification matrix are used to generate expert scoring results. According to the key anatomical structures segmented and located, the key anatomical structure co-occurrence matrix diagram, the blood flow classification matrix and the blood flow-key anatomical structure co-occurrence matrix diagram, a classification prediction result is generated. The expert scoring results, the classification prediction results and the diagnosis and treatment decisions entered by the medical staff are integrated to generate the final prediction results; the diagnosis and treatment decisions entered by the medical staff are based on the key anatomical structures segmented and located, the blood flow classification results, the expert scoring results and the classification prediction results, combined with the personal clinical diagnosis and treatment experience. The blood flow classification result is a natural language representation of the blood flow classification matrix.

[0098] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0099] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A cross-modal early pregnancy fetal congenital heart disease diagnosis system, characterized in that: include: A semantic segmentation module, used to segment and locate key anatomical structures of a standard section of early pregnancy fetal congenital heart disease from an input image; the key anatomical structures include a four-chamber heart standard section and a three-vessel trachea standard section; A matrix construction module is used to generate a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram according to the key anatomical structures located by segmentation; wherein the matrix construction module includes: a category interaction diagram calculation unit, used to calculate the category interaction diagram of the four-chamber heart standard section and the three-vessel trachea standard section according to the key anatomical structures located by segmentation; a dictionary matrix construction unit, used to construct a dictionary matrix according to the category values ​​of the key anatomical structures located by segmentation; a co-occurrence matrix diagram generation unit, used to generate the key anatomical structure co-occurrence matrix diagram of the four-chamber heart standard section and the three-vessel trachea standard section according to the dictionary matrix and the category interaction diagram; a blood flow classification matrix generation unit, used to predict the blood flow quantity, shape and color of the four-chamber heart standard section and the three-vessel trachea standard section according to the key anatomical structures located by segmentation, and generate a corresponding blood flow classification matrix; a blood flow-key anatomical structure co-occurrence matrix diagram generation unit, used to combine the blood flow classification matrix with the key anatomical structure co-occurrence matrix diagram to generate a blood flow-key anatomical structure co-occurrence matrix diagram; An expert knowledge scoring module is used to generate expert scoring results based on preset expert knowledge rules and by using the key anatomical structures and blood flow classification matrix located by segmentation; The congenital heart disease classification module is used to generate classification prediction results based on the key anatomical structures, key anatomical structure co-occurrence matrix diagrams and blood flow-key anatomical structure co-occurrence matrix diagrams located by segmentation; The artificial intelligence module is used to display the key anatomical structures located by segmentation, blood flow classification results, expert scoring results and classification prediction results, so that medical staff can enter corresponding diagnosis and treatment opinions and decisions; the blood flow classification results are natural language representations of the blood flow classification matrix; The result fusion module is used to integrate the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff to generate the final prediction results.

2. A cross-modality early pregnancy fetal congenital heart disease diagnosis system as claimed in claim 1, characterized in that: The semantic segmentation module is further used to fuse the input two-dimensional grayscale image and color Doppler image of the same section, and segment and locate the key anatomical structures required for standard section detection of fetal congenital heart disease in early pregnancy based on the image obtained by the fusion process.

3. A cross-modality early pregnancy fetal congenital heart disease diagnosis system as claimed in claim 1, characterized in that: The neural network structure of the blood flow classification matrix generation unit includes multiple fully connected layers, and calculates the prediction loss of blood flow quantity, shape and color through a Softmax cross entropy loss function.

4. A cross-modality early pregnancy fetal congenital heart disease diagnosis system as claimed in claim 1, characterized in that: The expert knowledge rules are scoring rules constructed based on the evaluation results of key features in key anatomical structures, including whether there is a single ventricle and asymmetry of color blood flow bundles between the left and right ventricles in the four-chamber heart standard section, whether only one vascular arch is displayed in the three-vessel trachea standard section, and whether there is abnormal U-shaped vascular arrangement.

5. A cross-modality early pregnancy fetal congenital heart disease diagnosis system as claimed in claim 1, characterized in that: The artificial intelligence module is further used to adjust the selection of the input image in response to the standard section selection instructions of the medical staff.

6. A cross-modality early pregnancy fetal congenital heart disease diagnosis system as claimed in claim 1, characterized in that: The result fusion module combines the expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by medical staff as training data, and uses the AdaBoost algorithm to pre-train a strong classifier model.

7. A method, characterized in that The following steps are involved: Segment and locate key anatomical structures of a standard section of early pregnancy fetal congenital heart disease from an input image; the key anatomical structures include a four-chamber heart standard section and a three-vessel trachea standard section; According to the key anatomical structures located by segmentation, a key anatomical structure co-occurrence matrix diagram, a blood flow classification matrix and a blood flow-key anatomical structure co-occurrence matrix diagram are generated; specifically, the following steps are performed: according to the key anatomical structures located by segmentation, a category interaction diagram of a four-chamber heart standard section and a three-vessel tracheal standard section is calculated; according to the category values ​​of the key anatomical structures located by segmentation, a dictionary matrix is ​​constructed; according to the dictionary matrix and the category interaction diagram, a key anatomical structure co-occurrence matrix diagram of a four-chamber heart standard section and a three-vessel tracheal standard section is generated; according to the key anatomical structures located by segmentation, the blood flow quantity, shape and color of the four-chamber heart standard section and the three-vessel tracheal standard section are predicted, and a corresponding blood flow classification matrix is ​​generated; the blood flow classification matrix is ​​combined with the key anatomical structure co-occurrence matrix diagram to generate a blood flow-key anatomical structure co-occurrence matrix diagram; According to the preset expert knowledge rules, the expert scoring results are generated using the key anatomical structures and blood flow classification matrix located by segmentation; Generate classification prediction results based on the key anatomical structures located by segmentation, the key anatomical structure co-occurrence matrix diagram, and the blood flow-key anatomical structure co-occurrence matrix diagram; The expert scoring results, classification prediction results and the diagnosis and treatment decisions entered by the medical staff are integrated to generate the final prediction results; the diagnosis and treatment decisions entered by the medical staff are the diagnosis and treatment decisions entered by the medical staff based on the key anatomical structures located by segmentation, blood flow classification results, expert scoring results and classification prediction results, combined with personal clinical diagnosis and treatment experience; the blood flow classification results are the natural language representation of the blood flow classification matrix.

8. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to claim 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 7 is implemented.

Images