Multimodal-based three-dimensional reconstruction and surgical planning method for fetal hydrocephalus

Abstract

The present application relates to the field of image processing, in particular to a fetal hydrocephalus three-dimensional reconstruction and operation planning method based on multi-modal, comprising: obtaining key points in three-dimensional point cloud and ventricle posture at each time of three-dimensional reconstruction, predicting ventricle posture at next time of three-dimensional reconstruction, and obtaining prediction residual; clustering the key points of historical three-dimensional reconstruction based on the prediction residual to obtain all categories; and based on the categories, further deleting and adding the obtained three-dimensional point cloud at the time of three-dimensional reconstruction again. The present application avoids the errors of matching and fusion process and the errors of prediction process at the time of three-dimensional reconstruction, and improves the accuracy of three-dimensional reconstruction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a fetal hydrocephalus three-dimensional reconstruction and surgery planning method based on multi-modal. BACKGROUND

[0002] Real-time three-dimensional reconstruction of the brain of the fetus in the mother can not only assist in judging the severity of fetal hydrocephalus, but also be used for planning the surgical scheme of hydrocephalus drainage. The commonly used three-dimensional reconstruction method is to collect ultrasound images or MRI images by using ultrasonic imaging technology or MRI imaging technology, and then to reconstruct a three-dimensional model of the brain or a three-dimensional point cloud of the internal tissue of the brain by using image matching and image fusion technology. In addition, since the posture of the fetus in the mother is variable and cannot be fixed, the scanning direction of the ultrasound probe and the MRI device needs to be predicted according to the historical posture time sequence of the ventricle during three-dimensional reconstruction, so as to adapt to the real-time changing posture of the fetus, so as to obtain accurate and reliable three-dimensional reconstruction results (such as three-dimensional point cloud).

[0003] However, there may be errors in the image matching and fusion process during three-dimensional reconstruction (for example, it is difficult to obtain more accurate and reliable corner point pairs), and there may also be errors in the prediction process of the scanning direction of the ultrasound probe and the MRI device (for example, due to too many corner point pairs in the matching and fusion process, the time spent in the two three-dimensional reconstruction processes is long, and thus the sampling frequency of the historical posture time sequence of the ventricle is too small, which cannot describe and reflect the changing trend of the posture of the fetus, and thus the prediction process has errors). The errors in the above two aspects make the three-dimensional point cloud obtained in the real-time three-dimensional reconstruction process unreliable (for example, the three-dimensional point cloud cannot accurately describe the boundary profile of the ventricle and the brain). SUMMARY

[0004] To solve the above problems, the present application provides a fetal hydrocephalus three-dimensional reconstruction and surgery planning method based on multi-modal.

[0005] The fetal hydrocephalus three-dimensional reconstruction and surgery planning method based on multi-modal of the present application adopts the following technical scheme:

[0006] One embodiment of the present application provides a fetal hydrocephalus three-dimensional reconstruction method based on multi-modal, which comprises the following steps:

[0007] The i-th three-dimensional reconstruction comprises: matching and fusing the ultrasound images and the MRI images to obtain a three-dimensional point cloud of the fetal ventricle, and obtaining a ventricle posture, wherein the point cloud obtained by the corner point matching is referred to as a key point; taking the corner point matching similarity of the key point and the distribution of the key point in the ventricle as a feature vector of the key point; predicting the ventricle posture according to the time sequence of the ventricle posture, and adjusting the scanning direction of the ultrasound probe according to the predicted ventricle posture;

[0008] performing the i+1th three-dimensional reconstruction according to the ultrasound image and the MRI image in the scanning direction;

[0009] In the i th and i+1th three-dimensional reconstruction process, the difference between the predicted residual error after predicting the ventricle posture and the length of the feature vector of all key points is recorded as a first difference; the feature vectors of all key points obtained in the historical three-dimensional reconstruction are clustered, and the number of categories obtained is negatively correlated with the first difference; the average length of the feature vectors of the key points in the category is recorded as an index of the category;

[0010] When performing the i+2th three-dimensional reconstruction process, the obtained key points are recorded as key points S; the key points S in the category are deleted in order from small to large according to the category index, or the key points S are added to the category in order from large to small according to the category index, so that the number of key points S in all categories is equal to N, and N is positively correlated with the first difference.

[0011] Preferably, the obtained key points are recorded as key points S; the key points S in the category are deleted in order from small to large according to the category index, or the key points S are added to the category in order from large to small according to the category index, so that the number of key points S in all categories is equal to N, and the specific steps include the following:

[0012] For each ultrasound image and the matching MRI image, the key points S obtained by performing corner point matching; when the number of key points S is greater than N; the key points S in the category are deleted in order from small to large according to the category index, so that the number of key points S in all categories is equal to N; when the number of key points S is less than N, the key points S are added to the category in order from large to small according to the category index, so that the number of key points S is equal to N.

[0013] Preferably, the key points S are added to the category in order from large to small according to the category index, and the specific steps include the following:

[0014] Each key point corresponds to an angle point pair obtained by angle point matching, and the angle points in the angle point pair corresponding to each key point in the category form a first attention area on the same ultrasound image; the angle points in the angle point pair corresponding to each key point in the category form a second attention area on the same MRI image;

[0015] In the first attention area and the second attention area corresponding to the category with the largest category index, the parameters of the corner detection algorithm are reset, and corner detection and corner matching are performed again, and the obtained angle point pair is regarded as a key point S and added to the category with the largest category index;

[0016] When the number of key points S is still less than N, corner detection and corner matching are performed again in the first and second attention areas corresponding to the second largest category of the category index. The resulting corner pairs are regarded as key points S and added to the second largest category of the category index.

[0017] This process continues until the number of keypoints S equals N. When the number of keypoints S is greater than N, keypoints S within each category are deleted in ascending order of category index, so that the number of keypoints S in all categories equals N.

[0018] Preferably, the corner point from the MRI image in the corner point pair corresponding to the key point S is denoted as the corner point to be classified, and the category corresponding to the second region of interest closest to the corner point to be classified is taken as the category to which the key point S belongs.

[0019] Preferably, the specific steps for deleting key points S within a category according to the category index in ascending order, so that the number of key points S in all categories is equal to N, are as follows:

[0020] If deleting several keypoints S in any category results in the total number of keypoints S across all categories being equal to N, then when deleting keypoints S in any category, they should be deleted sequentially in ascending order of their matching similarity.

[0021] Preferably, the ventricular pose is a vector formed by the center point of the ventricular three-dimensional point cloud and the directions of all principal components of the ventricular three-dimensional point cloud.

[0022] Preferably, the distribution of the key points in the ventricles is the maximum projected length of the key points in all principal component directions.

[0023] Preferably, the specific steps for calculating the prediction residual after predicting the ventricular posture are as follows:

[0024] The 3D point cloud of the ventricle obtained during the (i+1)th 3D reconstruction is denoted as the ventricle pose obtained from this 3D point cloud. The Euclidean distance between the (i+1)th ventricle pose and the predicted ventricle pose is denoted as the prediction residual after predicting the ventricle pose.

[0025] Preferably, the specific steps for obtaining a three-dimensional point cloud of the fetal ventricles by corner matching and fusion of ultrasound images and MRI images are as follows:

[0026] Corner point matching is performed between each ultrasound image and the corner point in the matching MRI image. The homography matrix is ​​obtained using the corner point pairs obtained by corner point matching. The matching MRI image is then affinely transformed using the homography matrix. The affinely transformed matching MRI image and ultrasound image are then fused into a fused image using the Laplacian pyramid fusion algorithm. Pixels in all fused images with gradient amplitudes greater than a first preset threshold are recorded as target pixels. All target pixels constitute a three-dimensional point cloud of the fetal ventricles.

[0027] Preferably, the specific steps for matching the MRI images are as follows:

[0028] Corner point matching is performed on any ultrasound image and any MRI image. For all corner point pairs obtained by corner point matching, the mean of the corner point matching similarity of all corner point pairs is recorded as the matching degree between the ultrasound image and the MRI image. Among the matching degrees between any ultrasound image and all MRI images, the MRI image with the highest matching degree is taken as the matching MRI image of any ultrasound image.

[0029] The beneficial effects of the technical solution of the present invention are:

[0030] In real-time three-dimensional reconstruction of the fetus, this invention ensures that the ultrasound probe and the MRI scanning plane are always aligned with the fetus during each reconstruction, thereby adapting to the fetus's posture in the womb in real time and preventing changes in the fetus's posture from affecting the three-dimensional reconstruction results (for example, preventing the point cloud obtained from the three-dimensional reconstruction from failing to represent the boundary contours of the ventricles and the brain).

[0031] In the process of adapting the fetus's posture in the womb, this invention can increase the accuracy of posture adaptation (i.e., reduce the prediction residual after predicting the ventricular posture). Specifically, this invention increases this accuracy in two ways: firstly, considering that low accuracy is caused by errors in the matching and fusion process of 3D reconstruction, and secondly, by errors in the posture prediction process. Based on this, the present invention divides the key points in the historical 3D reconstruction process into different categories through the first difference. Using this category, the key points (specifically key points S) obtained in the subsequent 3D reconstruction process (i.e., the i+2th 3D reconstruction process) are deleted and added. On the one hand, this avoids that too many key points S will lead to a longer time for the subsequent matching and fusion process, resulting in a low sampling frequency of the temporal sequence of the ventricle posture, which makes the temporal sequence unable to describe and reflect the changing trend of the fetal posture (especially when the change in fetal posture is too large within a period of time between two adjacent 3D reconstructions), ultimately leading to errors in the prediction process. On the other hand, it avoids that too few key points S will lead to a lack of matching brain tissue texture (or a lack of matching corner point pairs) in the subsequent 3D reconstruction process, resulting in inaccurate 3D point clouds that cannot describe the brain tissue structure, ultimately leading to errors in the matching and fusion results.

[0032] In summary, this invention increases the accuracy of adapting to fetal posture from the above two aspects, thereby comprehensively and accurately improving the accuracy of the three-dimensional reconstruction results.

[0033] In addition, the deletion and addition of key points S are based on the category index of the category to which the key point S belongs, which makes the remaining key points S able to describe the actual posture of the fetal ventricles in the mother's body relatively accurately.

[0034] In addition, the number of categories in this invention is negatively correlated with the first difference, while N is positively correlated with the first difference. When the first difference is larger, the final number of keypoints S, N, is larger, which helps reduce errors in subsequent matching and fusion. Simultaneously, the fewer the number of categories, the larger the total number of keypoints per category. Therefore, when adding keypoints S, the added keypoints not only have high matching similarity and projection length but also cover a wide range of diverse brain tissue textures, further ensuring the accuracy of subsequent matching and fusion results. Conversely, when the first difference is smaller, the final number of keypoints S, N, is smaller, which helps reduce errors in the prediction process. Simultaneously, the more categories, the smaller the total number of keypoints per category. This avoids spending excessive computation time adding keypoints S, further ensuring the accuracy of subsequent prediction processes. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating the steps of a multimodal three-dimensional reconstruction method for fetal hydrocephalus provided in an embodiment of the present invention. Detailed Implementation

[0037] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the multimodal-based three-dimensional reconstruction and surgical planning method for fetal hydrocephalus proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] The following description, in conjunction with the accompanying drawings, details the specific scheme of the multimodal fetal hydrocephalus-based three-dimensional reconstruction and surgical planning method provided by this invention.

[0040] Example 1:

[0041] Please see Figure 1 The diagram illustrates a flowchart of a multimodal three-dimensional reconstruction method for fetal hydrocephalus provided by an embodiment of the present invention. The method includes the following steps:

[0042] Step S101: Perform corner matching and fusion of ultrasound images and MRI images, and perform three-dimensional reconstruction to obtain a three-dimensional point cloud of the fetal ventricle. The point cloud obtained by corner matching is recorded as key points.

[0043] First, it should be noted that hydrocephalus originates in the ventricles. In order to obtain the real-time status (e.g., volume) of hydrocephalus in the fetus, it is necessary to reconstruct the fetal ventricles in real time in three dimensions.

[0044] Each 3D reconstruction process includes:

[0045] The ultrasound probe rapidly translates and scans along a direction parallel to the cranial-mandibular plane to obtain a three-dimensional ultrasound image (or a two-dimensional multi-planar ultrasound image). The three-dimensional ultrasound image contains several two-dimensional ultrasound images. For example, a two-dimensional ultrasound image is acquired every 1 mm of translation. Each ultrasound image contains cross-sectional information of the fetal brain.

[0046] During the first 3D reconstruction, the fetus's posture in the womb cannot be obtained, and therefore the craniotomy-mandibular plane cannot be determined. At this time, the doctor manually moves the ultrasound probe to observe the fetus's posture and find the craniotomy-mandibular plane of the fetus's head, as well as the area where the fetus's head is located. The ultrasound probe moves rapidly from left to right along the direction parallel to the craniotomy-mandibular plane to scan the head area, acquiring a 2D ultrasound image every 1mm of translation.

[0047] Furthermore, in this embodiment, two-dimensional multiplanar cross-sectional images of the fetal brain are obtained by scanning in the same direction using a fast MRI sequence (such as SSFSE, FIESTA), and each cross-sectional image is referred to as an MRI image.

[0048] Thus, two-dimensional multiplanar ultrasound images and two-dimensional multiplanar MRI images were obtained using ultrasound and MRI technologies, both of which are grayscale images.

[0049] In other embodiments, rapid translation scanning can be performed along other directions; this embodiment does not limit the specific scanning direction.

[0050] Ultrasound images and MRI images describing the same cranial cross-section are matched and fused (specifically, the MRI image is matched and fused into the ultrasound image) to obtain a two-dimensional multi-planar fused image. Pixels with a gradient amplitude greater than a first preset threshold th1 (e.g., th1=30) are extracted from each fused image and designated as target pixels. These target pixels represent fetal tissue structures (e.g., the boundary contours of the ventricles and brain, the vascular contours within the ventricles, etc.). All target pixels in the two-dimensional multi-planar fused image constitute a point cloud in three-dimensional space.

[0051] In this embodiment, the center point of the ultrasound image acquired by the ultrasound probe at the starting position (e.g., the leftmost part of the fetal brain) is taken as the origin, the horizontal and vertical directions of the ultrasound image are taken as the x-axis and y-axis, and the direction perpendicular to the ultrasound image is taken as the z-axis. The origin and the x-axis, y-axis and z-axis constitute the coordinate system of the point cloud.

[0052] Additionally, it should be noted that the method for obtaining the x-axis and y-axis coordinates of each pixel in an ultrasound image in a coordinate system is as follows:

[0053] To obtain the longitudinal and lateral distances of an ultrasound image, specifically: longitudinal distance (mm / pixel) = imaging depth (mm) / number of longitudinal pixels in the image; lateral distance (mm / pixel) = scan width (mm) / number of lateral pixels in the image;

[0054] The product of the number of horizontal pixels and the horizontal distance between each pixel and the center point of the ultrasound image is denoted as the x-axis coordinate, and the product of the number of vertical pixels and the vertical distance between each pixel and the center point of the ultrasound image is denoted as the y-axis coordinate.

[0055] The imaging depth and scanning width are obtained directly from the ultrasound probe, and will not be described in detail in this embodiment.

[0056] In other embodiments, other coordinate systems may be set, and this embodiment does not impose specific limitations.

[0057] Thus, a three-dimensional point cloud of the fetal head in the womb was obtained. In this embodiment, the PointNet neural network was used to segment the three-dimensional point cloud of the fetal ventricles from the three-dimensional point cloud of the head; the three-dimensional point cloud mentioned in this embodiment thereafter refers to the three-dimensional point cloud of the ventricles.

[0058] As an example, the matching and fusion of ultrasound images and MRI images includes the following methods:

[0059] This embodiment utilizes the SIFT corner detection algorithm to obtain all corners on each ultrasound and MRI image. The parameter settings for the SIFT corner detection algorithm are shown in the table below;

[0060] Table 1. Schematic diagram of parameter settings for the SIFT corner detection algorithm.

[0061]

[0062] The SIFT corner detection algorithm is a well-known technology, and will not be described in detail in this embodiment. In other embodiments, the above parameters of the algorithm can also be set to other values, and will not be described in detail in this embodiment.

[0063] In another embodiment, the Harris algorithm can be used to obtain all corner points on each ultrasound image and MRI image. The parameters of the Harris algorithm are set as shown in the table below.

[0064] Table 2. Schematic diagram of parameter settings for the Harris algorithm.

[0065]

[0066] The Harris algorithm is a well-known technique, and will not be described in detail in this embodiment. In other embodiments, the above parameters of the algorithm can also be set to other values, and will not be described in detail in this embodiment.

[0067] The corner points on any ultrasound image and any MRI image are matched to obtain several corner point pairs. In each corner point pair, one corner point comes from the ultrasound image and the other corner point comes from the MRI image. Each corner point pair corresponds to a corner point matching similarity (that is, the cosine similarity between corner point descriptors). The mean of the corner point matching similarity of all corner point pairs is recorded as the degree of matching between the ultrasound image and the MRI image.

[0068] The algorithm used in this embodiment for corner matching is the kd-tree matching algorithm.

[0069] Among any given ultrasound image and all MRI images, the MRI image with the highest matching degree is selected as the matched MRI image for that given ultrasound image. The matched MRI image is then fused into the ultrasound image to obtain a fused image.

[0070] As an example, fusing matching MRI images into ultrasound images to obtain a fused image includes the following methods:

[0071] For all corner pairs between the ultrasound image and the matching MRI image, a homography matrix is ​​calculated using these corner pairs. This homography matrix is ​​then used to perform an affine transformation on the pixels in the matching MRI image, aligning the matching MRI image with the ultrasound image. Finally, the Laplacian pyramid fusion algorithm is used to fuse the affine-transformed matching MRI image and the ultrasound image into a fused image.

[0072] It should be further noted that the three-dimensional point cloud obtained in this embodiment is composed of pixels on the fused image. Considering that some pixels on the fused image are obtained from corner pairs (specifically from the affine transformation of corner pairs), a portion of the point cloud in the three-dimensional point cloud corresponds to corner pairs (or a portion of the point cloud is obtained by matching and fusing corner pairs). In this embodiment, these point clouds are recorded as key points.

[0073] The homography matrix, affine transformation, and Laplace pyramid fusion mentioned above are all existing technologies, and will not be described in detail in this embodiment.

[0074] Step S102: The principal component direction of the 3D point cloud is used as the pose of the ventricle, and the corner matching similarity of the key points and the maximum projection length of the key points in the principal component direction are used as the feature vectors of the key points.

[0075] (1) Obtain the center point of all three-dimensional point clouds of the ventricle, and use the PCA algorithm to obtain all principal component directions of the three-dimensional point cloud of the ventricle. The vector formed by splicing the center point and all principal component directions end to end is used as the pose of the ventricle.

[0076] The PCA algorithm is a well-known technique that produces principal component vectors whose directions are mutually orthogonal.

[0077] (2) In this embodiment, each key point is obtained by matching and fusing a corner point pair, which corresponds to a corner point matching similarity (see step S101 for details). The larger the corner point matching similarity, the more accurate the texture (e.g., the texture represented by the tissue structure of the ventricle) at the key point is, which helps to ensure the accuracy of matching and fusing ultrasound images and MRI images (and better ensures the accuracy of the 3D point cloud and ventricle pose of the ventricle). The smaller the corner point matching similarity, the less conducive it is to the accurate matching and fusing of ultrasound images and MRI images.

[0078] Compared to keypoints, other point clouds outside of keypoints do not undergo texture matching (i.e., they are not obtained through corner point matching). Therefore, it cannot be guaranteed that identical textures will be merged during the fusion process. Consequently, the accuracy of point clouds outside of keypoints is relatively low, or in other words, they cannot accurately and reliably represent the tissue structures within the ventricles (e.g., they cannot accurately represent the boundary contours between the ventricles and the brain). Therefore, if ventricular pose is primarily represented and described based on point clouds outside of keypoints, the obtained ventricular pose will be inaccurate; conversely, if ventricular pose is primarily represented and described based on keypoints, the obtained ventricular pose will be relatively reliable.

[0079] Obtain the maximum projection length of each keypoint in all principal component directions. The larger the maximum projection length, the more accurate the ventricular posture is, as it is mainly reflected and described by keypoints.

[0080] (3) In this embodiment, the two-dimensional vector formed by the corner matching similarity of each key point and the maximum projection length of the key point in the principal component direction is denoted as the feature vector of each key point. The larger the magnitude of the feature vector of each key point (that is, the larger the corner matching similarity and the maximum projection length), the more accurate and reliable the ventricular posture is obtained. The distribution of the feature vectors of all key points can describe the overall accuracy of the ventricular posture. For example, the more and denser the distribution of key points with large magnitudes, the more accurate the ventricular posture acquisition result is. Otherwise, the ventricular posture obtained above may not be able to reliably and accurately describe the true posture of the fetal ventricles in the mother's body.

[0081] Step S103: Predict the ventricular posture based on the temporal sequence of the ventricular posture, and adjust the ultrasound scanning direction according to the predicted ventricular posture.

[0082] This embodiment requires real-time 3D reconstruction of the fetal ventricles in the womb; therefore, the 3D reconstruction process in step S101 needs to be executed multiple times. Assuming that the 3D reconstruction process in step S101 represents the i-th 3D reconstruction process, the ventricle poses obtained from the i-th 3D reconstruction process and all previous 3D reconstruction processes constitute a temporal sequence. This embodiment uses a temporal prediction algorithm to predict this temporal sequence, obtaining the predicted ventricle pose for the next (i+1) 3D reconstruction process.

[0083] It should be noted that the predicted ventricular posture is obtained by translating and rotating the ventricular posture obtained in the i-th time. These two transformations represent the changes in the fetal head posture in the mother's body.

[0084] This embodiment uses the ventricular pose obtained in the i-th iteration and the predicted ventricular pose to solve for the translation and rotation. The translation is the displacement vector from the center point in the i-th obtained ventricular pose to the center point in the predicted ventricular pose. The rotation is the rotation matrix when rotating from all principal component directions of the i-th obtained ventricular pose to all principal component directions of the predicted ventricular pose. The calculation methods for the displacement vector and rotation matrix are well-known and will not be elaborated upon in this embodiment.

[0085] Then the ultrasound probe also changes position based on the translation and rotation (that is, it translates according to the displacement vector and rotates according to the rotation matrix), and then performs a rapid translation scan again to obtain a three-dimensional ultrasound image (that is, a two-dimensional multiplanar ultrasound image). The rapid MRI sequence technology is used to obtain a two-dimensional multiplanar MRI image of the fetal brain, and then the point cloud is reconstructed in three dimensions using the method in step S101, so as to obtain the three-dimensional point cloud of the fetal ventricle at the i+1th three-dimensional reconstruction.

[0086] In step S104, during the i-th and i+1-th three-dimensional reconstruction processes, the difference between the prediction residual after predicting the ventricle pose and the modulus of the feature vectors of all key points is denoted as the first difference.

[0087] The ventricular pose is obtained using the 3D point cloud of the fetal ventricle obtained during the (i+1)th 3D reconstruction (referred to as the (i+1)th ventricular pose). The difference between this ventricular pose and the predicted ventricular pose obtained in step S103 (e.g., the Euclidean distance between the two) is denoted as the prediction residual after predicting the ventricular pose.

[0088] The larger the prediction residual, the less accurate the predicted ventricular pose is after the i-th 3D reconstruction. This inaccuracy may be due to two main reasons. First, it may be caused by inaccuracies in the matching and fusion results during the i-th and (i+1)-th 3D reconstructions. For example, if the 3D reconstruction process does not rely on more matching brain tissue textures (or corner point pairs), the 3D point cloud will be inaccurate and unable to describe the brain tissue structure (e.g., most of the obtained point cloud cannot describe the outline of the ventricles), ultimately leading to errors in the matching and fusion results. Second, it may be due to errors in the prediction process itself. For example, if each 3D reconstruction process takes a long time (e.g., due to the large number of corner point pairs obtained from multiple ultrasound and MRI images, the matching and fusion process will take a long time), the temporal sequence sampling frequency of the ventricular pose will be low. This temporal sequence cannot describe and reflect the changing trend of fetal pose (especially when the change in fetal pose is too large within a period of time between two adjacent 3D reconstructions), ultimately leading to errors in the prediction process.

[0089] In summary, errors in the matching and fusion results, as well as errors in the prediction process, can lead to larger prediction residuals and less accurate predictions of ventricular posture. This results in the ultrasound probe's orientation and position not being able to match the fetal posture, which in turn leads to inaccuracies in the subsequent 3D reconstruction process (for example, the point cloud obtained in the subsequent 3D reconstruction process cannot accurately represent the boundary contours of the ventricles and the brain).

[0090] Furthermore, for all key points obtained in the i-th and i+1-th 3D reconstruction processes, the average magnitude of the feature vectors of these key points is denoted as the matching and fusion accuracy of the i-th and i+1-th 3D reconstruction processes. The larger this value, the greater the accuracy of matching and fusion of the ultrasound images and MRI images included in the i-th and i+1-th processes, and the smaller the error of the matching and fusion result; conversely, the smaller this value, the smaller the accuracy of matching and fusion of the ultrasound images and MRI images included in the i-th and i+1-th processes, and the larger the error of the matching and fusion result.

[0091] The difference between the prediction residual and the accuracy of the matching fusion is denoted as the first difference. The larger this first difference, the smaller the accuracy of the matching fusion relative to the prediction residual. In this case, the error of the matching fusion result is greater than the error of the prediction process, meaning that not more corner pairs were used for matching fusion, resulting in inaccurate 3D point clouds. In this case, more corner pairs (or key points) need to be added. Conversely, the smaller the first difference, the larger the accuracy of the matching fusion relative to the prediction residual. In this case, the error of the prediction process is greater than the error of the matching fusion result, meaning that too many corner pairs and long matching fusion times result in a low sampling frequency of the temporal sequence of ventricular posture, leading to poor prediction results. In this case, the introduction of fewer corner pairs (or key points) needs to be reduced.

[0092] Based on this, this embodiment uses the first difference to limit the number of corner points acquired in the three-dimensional reconstruction process, so as to ensure the accuracy of the predicted ventricular pose (that is, to avoid excessive prediction residuals after predicting the ventricular pose).

[0093] As an example, methods for obtaining the prediction residuals after predicting ventricular posture include:

[0094] The Euclidean distance between the (i+1)th ventricular pose and the predicted ventricular pose is used as the prediction residual.

[0095] As an optional example, methods for obtaining the first difference include:

[0096] Let w1×x2-w2×x1 be denoted as the first difference, where x1 represents the matching fusion accuracy, x2 represents the prediction residual, and w1 and w2 represent two preset parameters. In this embodiment, w1=1.5 and w2=1 are used as examples. In other embodiments, they can be set to other values.

[0097] As a preferred example, methods for obtaining the first difference include:

[0098] Will Let 'first difference' be denoted as exp(), which represents an exponential function with the natural constant as its base.

[0099] Step S105: Cluster the feature vectors of all key points obtained in the historical 3D reconstruction. The number of categories is negatively correlated with the first difference. The average magnitude of the feature vectors of key points within a category is recorded as the category index.

[0100] For keypoints obtained in the (i+1)th and earlier 3D reconstructions (including the (i+1)th reconstruction), these keypoints describe the ventricular tissue present in the ventricles during the historical 3D reconstruction process, which can be relatively accurately matched on ultrasound and MRI images. Even if there are errors in the historical 3D reconstruction process, these keypoints can still serve as a reference for obtaining keypoints in subsequent 3D reconstructions.

[0101] The feature vectors of all key points obtained from the historical 3D reconstruction are clustered (this embodiment uses the K-Means algorithm for clustering). The number of clusters is negatively correlated with the first difference. The average magnitude of the feature vectors of key points within each cluster is recorded as the index of each cluster. This process ensures fewer clusters when the first difference is larger and more clusters when the first difference is smaller.

[0102] The larger the category index, the higher the priority of preserving the corner point pairs corresponding to the key points in that category, or in other words, the more desired it is to add corner point pairs in the category with the larger category index during the subsequent 3D reconstruction process.

[0103] As an example, the method for obtaining the number of categories is as follows:

[0104] Let exp(-f) × Q1 be the number of categories (rounded up), where exp() represents an exponential function with the natural constant as the base; f represents the first difference; and Q1 represents the preset first parameter. This embodiment uses Q1=20 as an example; other embodiments can be set to other values, and this embodiment does not impose specific limitations. Specifically, when the number of categories is greater than or equal to 20, or less than or equal to 3, let the number of categories be equal to 20 or 3.

[0105] Each keypoint within each category corresponds to a pair of corner points. One corner point in the pair comes from a pixel in a certain ultrasound image. That is, some keypoints within each category correspond one-to-one with pixels in the same ultrasound image. For all keypoints within each category, the keypoints of the corresponding pixels in the same ultrasound image are obtained. The region formed by the pixels corresponding to these keypoints in the same ultrasound image is called the first region of interest for each category in the ultrasound image.

[0106] Similarly, each keypoint within each category corresponds to a corner point pair, and the other corner point in the corner point pair comes from a pixel in a certain MRI image. That is, some keypoints within each category correspond one-to-one with pixels in the same MRI image. For all keypoints within each category, the keypoints of the corresponding pixels in the same MRI image are obtained. The region formed by the corresponding pixels of these keypoints in the same MRI image is called the second region of interest for each category in the MRI image.

[0107] Note that the sizes of ultrasound and MRI images remain unchanged in the historical 3D reconstruction.

[0108] Specifically, if the feature vectors of all key points obtained in the historical 3D reconstruction are less than 100, the subsequent steps will not be executed. Instead, 3D reconstruction will be performed using steps S101-S103.

[0109] In addition, in order to eliminate the order-of-magnitude differences between different dimensions of the feature vectors of key points, this embodiment performs whitening processing (e.g., ZCA whitening processing) on ​​the feature vectors of all key points before clustering.

[0110] Step S106: During the (i+2)th 3D reconstruction process, the obtained key points are recorded as key points S. Key points S in each category are deleted in ascending order of category index, or key points S are added in each category in descending order of category index, so that the number of key points S in all categories is equal to N, and N is positively correlated with the first difference.

[0111] During the (i+2)th three-dimensional reconstruction, all matching corner point pairs between each ultrasound image and the matching MRI image are obtained according to the method in step S101. Each corner point pair will subsequently correspond to a key point (that is, each corner point pair can be regarded as a key point), denoted as key point S.

[0112] During the (i+2)th three-dimensional reconstruction, several key points S were obtained on each ultrasound image, and these key points S belong to different categories.

[0113] In this step, key points S obtained from each ultrasound image during the (i+2)th 3D reconstruction are deleted and added to ensure that the final number of key points S equals N. This avoids two problems: First, too many key points S would lead to excessive time spent on subsequent matching and fusion processes, resulting in a low sampling frequency for the temporal sequence of ventricular posture, which would be unable to describe and reflect the changing trends of fetal posture (especially when the change in fetal posture is too large between two adjacent 3D reconstructions), ultimately leading to errors in the prediction process. Second, too few key points S would prevent the subsequent 3D reconstruction process from relying on more matching brain tissue textures (or corner point pairs) for matching and fusion, resulting in inaccurate 3D point clouds that cannot describe the brain tissue structure (e.g., most of the obtained point clouds cannot describe the outline of the ventricles), ultimately leading to errors in the matching and fusion results. Furthermore, the deletion and addition of key points S are based on the category index of the key point S's category, ensuring that the remaining key points S can relatively accurately describe the actual posture of the fetal ventricles in the womb.

[0114] Specifically, for each key point S (i.e., corner pair) obtained from an ultrasound image and a matching MRI image, and the category to which these key points S belong, the following processing is performed:

[0115] When the number of keypoints S is greater than N, keypoints S within each category are deleted in ascending order of category index, so that the number of keypoints S equals N. It should be noted that if deleting only some keypoints S within a category is sufficient to make the remaining number of keypoints S equal to N, then keypoint deletion within that category is performed randomly; in other embodiments, keypoints S can also be deleted sequentially within the same category in ascending order of matching similarity.

[0116] When the number of these keypoints S is less than N, keypoints S are added to the category in descending order of the category index, so that the final number of keypoints S is equal to N.

[0117] It should be noted that, since the number of categories is negatively correlated with the first difference, the larger the first difference (at which point the final number of keypoints S N is larger, in order to reduce the error of subsequent matching and fusion, as described in step S104), the fewer the number of categories, the larger the total number of keypoints contained in each category, and the larger the first and second regions of interest corresponding to each category. Therefore, when adding keypoints S, it can be ensured that keypoints are added in the first region of interest (in ultrasound images) and the second region of interest (in MRI images) with larger category indices and larger areas. This makes the added keypoints not only have a large matching similarity and projection length, but also cover a large range of diverse brain tissue textures, thereby further ensuring the accuracy of subsequent matching and fusion results.

[0118] The smaller the first difference (at which point the final number of key points S N is smaller, so as to reduce the error in the prediction process, as described in step S104), the more categories there are. Therefore, the total number of key points contained in each category is smaller, and the smaller the first and second regions of interest corresponding to each category, the more likely it is that when adding key points S, it can ensure that key points are added in the smaller areas of the first region of interest (in ultrasound images) and the second region of interest (in MRI images), avoiding the problem of spending more calculation time to add key points, and further ensuring the accuracy of the subsequent prediction process.

[0119] At this point, during the (i+2)th three-dimensional reconstruction process in this step, by deleting and adding key points S, the number of key points S obtained from each ultrasound image and the matching MRI image is equal to N.

[0120] As an optional example, N can be obtained as follows:

[0121] Let N = Q2 × f, where f represents the first difference and Q2 represents the preset second parameter. In this embodiment, Q2 = 20 is used as an example. In other embodiments, Q2 can be set to other values. In particular, when N is greater than or equal to 30, or less than or equal to 5, let N equal to 50 or 5. Note that N needs to be rounded up.

[0122] As a preferred example, N is obtained as follows:

[0123] The number of all corner pairs obtained between all ultrasound images and the matched MRI images in the historical 3D reconstruction process is obtained. The ratio of this number to the number of all ultrasound images is denoted as the average number of corner pairs. The average number of corner pairs is taken as the value of Q2, and N = Q2 × f is set.

[0124] As an example, the method for obtaining the category to which each keypoint S belongs is as follows:

[0125] Considering that the texture in MRI images is clearer than that in ultrasound images, in this embodiment, corner points from the MRI image within each keypoint S are designated as corner points to be classified. Additionally, the MRI image contains second regions of interest from different categories. The second region of interest closest to the corner point to be classified (specifically, the outline edge of the second region of interest is closest to the corner point) is obtained, and the category corresponding to this second region of interest is taken as the category to which the keypoint S belongs.

[0126] As an example, adding keypoints S within a category according to its category metrics from largest to smallest involves the following steps:

[0127] (1) Obtain the category with the largest category index.

[0128] (2) Obtain the first and second regions of interest for this category in each ultrasound image and the matching MRI image. Only in these two regions should corner detection be performed again. Note that when performing corner detection again, the parameters of the corner detection algorithm need to be set to other values ​​so as to obtain an equal number of corners in these two regions.

[0129] The parameters for the SIFT corner detection algorithm have been reset as shown in the table below;

[0130] Table 3. Schematic diagram of SIFT corner detection algorithm resetting

[0131]

[0132] This embodiment does not limit the results of resetting the corner detection algorithm parameters.

[0133] (3) For the corner points that are re-obtained in the above two regions (removing the existing corner points), the corner points in the two regions are matched, and all the resulting corner point pairs are regarded as key points S. These key points S are added to the category.

[0134] (4) Determine if the total number of key points S is equal to N. If it is still less than N, obtain the second largest category in the category index, and then re-execute (2) and (3) in this step. Continue in this manner until the total number of key points S is equal to N.

[0135] If the total number of keypoints S is greater than N, then all keypoints S in each category are deleted in ascending order of category index, so that the number of keypoints S equals N.

[0136] In particular, if the total number of key points S is still less than N after all categories have been processed according to (2) and (3) in this step, then the number of key points S is no longer required to be equal to N, and the subsequent process continues.

[0137] At this point, during the (i+2)th 3D reconstruction process, by deleting and adding key points S, the number of key points S obtained from each ultrasound image and the matching MRI image is equal to N (except in special cases). Next, all the remaining key points S are used to match and fuse the ultrasound and MRI images, and a 3D point cloud of the fetal ventricles is reconstructed (see S101 for details). Then, the ventricle pose is predicted (i.e., steps S102 and S103 are executed), thus ending the (i+2)th 3D reconstruction process. Then, the key points S obtained from the (i+3)th 3D reconstruction process are deleted and added using the methods in steps S104 to S106.

[0138] In summary, this embodiment utilizes the i-th and i+1-th 3D reconstruction processes to perform the i+2-th 3D reconstruction process. Subsequently, the method described in all the steps of this embodiment is used to perform the i+3-th 3D reconstruction process based on the i+1-th and i+2-th 3D reconstruction processes, and so on, to achieve real-time 3D reconstruction of the fetus.

[0139] This concludes the example.

[0140] In the process described above in this embodiment, when performing real-time three-dimensional reconstruction of the fetus, the ultrasound probe and the MRI scanning plane are always aligned with the same position of the fetus in each reconstruction, so as to adapt to the fetus's posture in the mother's body in real time and avoid changes in the fetus's posture that may affect the three-dimensional reconstruction results (for example, to avoid the point cloud obtained from the three-dimensional reconstruction from being unable to represent the boundary contour of the ventricles and the brain).

[0141] In the process of adapting the fetus to its position in the womb, this embodiment can increase the accuracy of the position adaptation (that is, it can reduce the prediction residual after predicting the ventricular position). Specifically, this embodiment increases accuracy in two ways: Firstly, considering that low accuracy is caused by errors in the matching and fusion process, and secondly by errors in the pose prediction process, this embodiment divides key points in the historical 3D reconstruction process into different categories through a first difference. This category is used to delete and add key points (specifically, key points S) obtained in subsequent 3D reconstruction processes (i.e., the i+2th 3D reconstruction process). This avoids excessive key points S, which would lead to longer matching and fusion processes and consequently, a lower sampling frequency for the temporal sequence of ventricular pose, making the temporal sequence unable to describe and reflect the changing trend of fetal pose (especially when the change in fetal pose is too large between two adjacent 3D reconstructions), ultimately resulting in errors in the prediction process. Secondly, it avoids insufficient key points S, which would prevent subsequent 3D reconstruction processes from using more matching brain tissue textures (or less corner point pairs) for matching and fusion, resulting in inaccurate 3D point clouds that cannot describe the brain tissue structure (e.g., most of the obtained point clouds cannot describe the outline of the ventricles), ultimately leading to errors in the matching and fusion results.

[0142] In summary, this embodiment improves the accuracy of adapting to fetal posture from the above two aspects, thereby comprehensively and accurately improving the accuracy of the three-dimensional reconstruction results.

[0143] In addition, the deletion and addition of key points S are based on the category index of the category to which the key point S belongs, which makes the remaining key points S able to describe the actual posture of the fetal ventricles in the mother's body relatively accurately.

[0144] In addition, in this embodiment, the number of categories is negatively correlated with the first difference, while N is positively correlated with the first difference. When the first difference is larger, the final number of keypoints S, N, is larger, which helps reduce errors in subsequent matching and fusion. Simultaneously, the number of categories is smaller, so each category contains a larger total number of keypoints. Therefore, when adding keypoints S, the added keypoints not only have high matching similarity and projection length but also cover a wider range of diverse brain tissue textures, further ensuring the accuracy of subsequent matching and fusion results. Conversely, when the first difference is smaller, the final number of keypoints S, N, is smaller, which helps reduce errors in the prediction process. Simultaneously, the number of categories is larger, so each category contains a smaller total number of keypoints. Adding keypoints S avoids the problem of spending excessive computation time on adding keypoints, further ensuring the accuracy of subsequent prediction processes.

[0145] It is worth noting that, in this embodiment, although the accuracy of adapting to the fetal posture was low in the first few (e.g., the first 5) 3D reconstructions, making it impossible to align the craniotop-mandibular plane for ultrasound and MRI image acquisition, even if the accuracy improved in subsequent 3D reconstructions, it still could not align the craniotop-mandibular plane again. However, this embodiment does not focus on whether the aligned plane is the craniotop-mandibular plane; as long as the accuracy is improved in subsequent reconstructions to ensure that the same plane is aligned each time (even if it is not the craniotop-mandibular plane), the accuracy of the 3D reconstruction results can still be guaranteed.

[0146] Example 2:

[0147] In the first embodiment, during the (i+2)th 3D reconstruction, it is necessary to first obtain key points S, and then delete and add key points S. This process still requires corner detection and corner matching of the entire ultrasound image or MRI image before obtaining key points S, and these processes will also take some computation time.

[0148] This embodiment provides a method for obtaining key points S and saving some of the calculation time for corner detection and corner matching, including:

[0149] During the (i+2)th 3D reconstruction, obtain the N2 categories with the largest index categories. For example, N2 is equal to half the number of categories (rounded up).

[0150] For all first and second regions of interest corresponding to these N2 categories, and for each ultrasound image and each MRI image, corner detection and matching are performed only within the first and second regions of interest corresponding to the N2 categories on the ultrasound and MRI images, thereby obtaining key points S for each ultrasound image and the matching MRI image. Then, it is determined whether the number of key points S obtained for each ultrasound image is greater than or equal to N. If it is greater than or equal to N, corner points in other regions on the ultrasound and MRI images are no longer acquired, nor are corner points in other regions matched with corner points in the MRI image. If it is less than N, corner detection and matching are performed within the first and second regions of interest corresponding to all categories. Then, it is determined whether the number of key points S obtained for each ultrasound image is greater than or equal to N. If it is greater than or equal to N, corner points in other regions on the ultrasound and MRI images are no longer acquired, nor are corner points in other regions matched with corner points in the MRI image. If the number of key points S obtained for each ultrasound image is still less than N, detection and corner matching are performed in all regions on the ultrasound and MRI images, thereby obtaining all key points S.

[0151] Next, following the method of step S106 in Example 1, all key points S are deleted and added, so that the number of key points S is equal to N.

[0152] Compared to Embodiment 1, this embodiment sacrifices the accuracy of the corner matching process (because corner detection and intersection matching are only performed in a local area), but it has a faster calculation speed and less computational load.

[0153] Example 3:

[0154] When the area of ​​the first or second region of interest in each category on each ultrasound or MRI image is small in Examples 1 and 2, or when the error in the historical 3D reconstruction process is large (e.g., when the prediction error and matching fusion error are large), it is impossible to obtain relatively accurate and reliable corner points within the first or second region of interest. In this case, it is necessary to expand the first or second region of interest.

[0155] The methods for expanding the first or second area of ​​concern are as follows:

[0156] The gray values ​​of pixels in the first or second region of interest corresponding to all categories are set to 1, and the gray values ​​of other pixels are set to 0, thereby generating a binary image for each ultrasound or MRI image. Each binary image is then expanded using a dilation operator, and the region in the expanded binary image (i.e., the region with a gray value of 1) is used as the expanded first or second region of interest corresponding to each category.

[0157] In this embodiment, the size of the dilation operator is 11×11, and all values ​​within the dilation operator are 1.

[0158] In some embodiments, the above-described dilation process is repeated multiple times, so that the gray value of all pixels in the region of the dilated binary image is 1. At this time, each ultrasound image or MRI image contains only different types of first or second regions of interest, and there are no regions outside the first or second regions of interest.

[0159] Example 4:

[0160] This embodiment provides a multimodal surgical planning method for fetal hydrocephalus, including: the doctor determines the location and volume of the ventricles based on real-time acquired three-dimensional reconstruction results, and performs puncture and drainage accordingly, such as draining to the outside of the mother's body, or the doctor obtains a drainage path based on the three-dimensional reconstruction results to facilitate the implantation of a drainage tube to drain the hydrocephalus into the mother's abdominal cavity. Specific puncture and drainage procedures are well-known techniques and will not be described in detail in this embodiment.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method for fetal hydrocephalus based on multimodal approaches, characterized in that, The method includes the following steps: The i-th three-dimensional reconstruction includes: performing corner matching and fusion of ultrasound images and MRI images to obtain a three-dimensional point cloud of the fetal ventricle and obtain the ventricle pose, wherein the point cloud obtained by corner matching is denoted as key points; using the corner matching similarity of key points and the distribution of key points in the ventricle as feature vectors of key points; predicting the ventricle pose based on the temporal sequence of the ventricle pose; and adjusting the scanning direction of the ultrasound probe based on the predicted ventricle pose. The (i+1)th three-dimensional reconstruction is performed based on the ultrasound and MRI images under the scanning direction; In the i-th and i+1-th 3D reconstruction processes, the difference between the prediction residual after predicting the ventricle pose and the modulus of the feature vectors of all key points is denoted as the first difference; the feature vectors of all key points obtained in the historical 3D reconstruction are clustered, and the number of categories obtained is negatively correlated with the first difference; the average modulus of the feature vectors of key points within a category is denoted as the category index. During the (i+2)th 3D reconstruction process, the obtained key points are denoted as key points S. Key points S in each category are deleted in ascending order of category index, or key points S are added to each category in descending order of category index, so that the number of key points S in all categories is equal to N, and N is positively correlated with the first difference.

2. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 1, characterized in that, The obtained key points are denoted as key points S. Key points S within each category are deleted sequentially according to the category index from smallest to largest, or key points S are added to each category in descending order of category index, so that the total number of key points S in all categories equals N. The specific steps are as follows: For each ultrasound image and the matching MRI image, key points S are obtained by corner matching. If the number of key points S is greater than N, key points S in each category are deleted in ascending order of category index, so that the number of key points S in all categories is equal to N. If the number of key points S is less than N, key points S are added to each category in descending order of category index, so that the number of key points S is equal to N.

3. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 2, characterized in that, The specific steps involved in adding key points S to the category according to the category index in descending order are as follows: Each key point corresponds to a corner point pair obtained by corner point matching. The area formed by the corner points of the corner point pair corresponding to the key point in each category on the same ultrasound image is called the first region of interest; the area formed by the corner points of the corner point pair corresponding to the key point in each category on the same MRI image is called the second region of interest. Within the first and second attention regions corresponding to the category with the largest category index, the parameters of the corner detection algorithm are reset, and corner detection and corner matching are performed again. The resulting corner pairs are regarded as key points S and added to the category with the largest category index. When the number of key points S is still less than N, corner detection and corner matching are performed again in the first and second attention areas corresponding to the second largest category of the category index. The resulting corner pairs are regarded as key points S and added to the second largest category of the category index. This process continues until the number of keypoints S equals N. When the number of keypoints S is greater than N, keypoints S within each category are deleted in ascending order of category index, so that the number of keypoints S in all categories equals N.

4. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 3, characterized in that, The corner point from the MRI image in the corner point pair corresponding to the key point S is denoted as the corner point to be classified, and the category corresponding to the second region of interest closest to the corner point to be classified is the category to which the key point S belongs.

5. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 2 or 3, characterized in that, The specific steps involved in deleting key points S within a category according to the category index in ascending order, so that the number of key points S in all categories equals N, are as follows: If deleting several keypoints S in any category results in the total number of keypoints S across all categories being equal to N, then when deleting keypoints S in any category, they should be deleted sequentially in ascending order of their matching similarity.

6. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 1, characterized in that, The ventricular pose is a vector formed by the center point of the ventricular 3D point cloud and the directions of all principal components of the ventricular 3D point cloud.

7. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 6, characterized in that, The distribution of keypoints in the ventricles is the maximum projection length of the keypoints in all principal component directions.

8. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 6, characterized in that, The specific steps for calculating the prediction residual after predicting the ventricular posture are as follows: The 3D point cloud of the ventricle obtained during the (i+1)th 3D reconstruction is denoted as the ventricle pose obtained from this 3D point cloud. The Euclidean distance between the (i+1)th ventricle pose and the predicted ventricle pose is denoted as the prediction residual after predicting the ventricle pose.

9. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodal approaches according to claim 8, characterized in that, The specific steps involved in obtaining a three-dimensional point cloud of the fetal ventricles by performing corner matching and fusion of ultrasound and MRI images are as follows: Corner point matching is performed between each ultrasound image and the corner point in the matching MRI image. The homography matrix is ​​obtained using the corner point pairs obtained by corner point matching. The matching MRI image is then affinely transformed using the homography matrix. The affinely transformed matching MRI image and ultrasound image are then fused into a fused image using the Laplacian pyramid fusion algorithm. Pixels in all fused images with gradient amplitudes greater than a first preset threshold are recorded as target pixels. All target pixels constitute a three-dimensional point cloud of the fetal ventricles.

10. The method for three-dimensional reconstruction of fetal hydrocephalus based on multimodality according to claim 2 or 9, characterized in that, The specific steps for obtaining the matched MRI images are as follows: Corner point matching is performed on any ultrasound image and any MRI image. For all corner point pairs obtained by corner point matching, the mean of the corner point matching similarity of all corner point pairs is recorded as the matching degree between the ultrasound image and the MRI image. Among the matching degrees between any ultrasound image and all MRI images, the MRI image with the highest matching degree is taken as the matching MRI image of any ultrasound image.

Images