An ultrasound image three-dimensional reconstruction method fusing multi-modal features

Abstract

The application discloses a kind of fusion multimodal feature's ultrasonic image three-dimensional reconstruction method, comprising the following steps: obtaining multimodal ultrasonic image data;Multi-modal feature extraction is carried out to multimodal ultrasonic image data, obtain multimodal feature set, and multimodal feature set includes structural feature set, texture feature set and functional feature set;Multi-modal feature set is registered and fused, and the three-dimensional feature atlas after fusion is obtained;The three-dimensional reconstruction model of ultrasonic image is generated by three-dimensional reconstruction processing to the three-dimensional feature atlas after fusion.The application has the following advantages and effects: by the synergistic fusion and space-time consistency optimization of multimodal feature, high-precision three-dimensional model that retains structure, texture and functional attribute simultaneously can be generated, and the clinical applicability and diagnostic value of reconstruction result are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultrasonic imaging, in particular to a three-dimensional reconstruction method of ultrasonic images fusing multi-modal features. BACKGROUND

[0002] The three-dimensional reconstruction technology of ultrasonic images aims to construct a three-dimensional model with stereoscopic morphology and internal features from a series of two-dimensional ultrasonic images, to provide more intuitive and accurate visual basis for clinical diagnosis, surgical planning and interventional treatment. At present, most of the methods of this kind are based on image sequences of a single imaging mode (such as B-mode ultrasound) for reconstruction, mainly relying on morphological structure or gray texture information to realize three-dimensional space mapping.

[0003] The core problem in the prior art is that there is a lack of effective feature-level fusion mechanism between multi-modal ultrasonic image data, which makes it difficult for the three-dimensional reconstruction model to simultaneously retain structural features, texture features and functional features. Specifically, different imaging modes differ in acquisition time, spatial resolution and feature representation dimension, and if only simple image alignment or superposition is performed, feature conflicts, spatial inconsistency and information loss are easily caused, thereby limiting the value of the reconstructed model in clinical applications. Therefore, there is an urgent need for a three-dimensional reconstruction method that can cooperatively fuse multi-modal features and maintain their spatio-temporal consistency. SUMMARY

[0004] The purpose of the present application is to provide a three-dimensional reconstruction method of ultrasonic images fusing multi-modal features to solve the problems raised in the background art.

[0005] The above technical purpose of the present application is achieved by the following technical solution:

[0006] A three-dimensional reconstruction method of ultrasonic images fusing multi-modal features, comprising the following steps:

[0007] S100, acquiring multi-modal ultrasonic image data; wherein the multi-modal ultrasonic image data comprises a sequence of ultrasonic images from different imaging modes, and each frame of image in the multi-modal ultrasonic image data is provided with an image acquisition time stamp and an imaging mode identifier;

[0008] S200, performing multi-modal feature extraction on the multi-modal ultrasonic image data to obtain a multi-modal feature set; wherein the multi-modal feature set comprises a structural feature set, a texture feature set and a functional feature set, the structural feature set is used to describe anatomical structure information in the ultrasonic image, the texture feature set is used to describe tissue texture information in the ultrasonic image, and the functional feature set is used to describe blood flow dynamics and tissue mechanics attribute information in the ultrasonic image;

[0009] S300, performing multi-modal feature registration and fusion on the multi-modal feature set to obtain a fused three-dimensional feature graph; wherein the fused three-dimensional feature graph comprises a spatially registered structural feature graph, a texture feature graph and a functional feature graph, and each feature point in the fused three-dimensional feature graph has spatial coordinate information and feature weight;

[0010] S400, performing three-dimensional reconstruction processing on the fused three-dimensional feature graph to generate a three-dimensional reconstruction model of the ultrasound image; wherein the three-dimensional reconstruction processing comprises voxelization processing, feature weighted fusion, isosurface extraction and mesh optimization.

[0011] By adopting the above technical solutions, by acquiring multi-modal ultrasound image data and extracting a multi-modal feature set including a structural feature set, a texture feature set and a functional feature set, anatomical structure information, tissue texture information and blood flow dynamics or tissue mechanics attribute information in the ultrasound image can be comprehensively captured, thereby providing a rich and multi-dimensional feature basis for three-dimensional reconstruction; by performing registration and fusion on the multi-modal feature set to obtain a fused three-dimensional feature graph, the consistency and complementarity of the features in space are ensured, and the accuracy and integrity of the reconstruction model are improved; finally, by performing three-dimensional reconstruction processing to generate a three-dimensional reconstruction model of the ultrasound image, the final model not only has a high-precision spatial structure but also reflects the functional attributes of the tissue, which helps doctors to make more accurate diagnosis and treatment planning, and the overall method improves the comprehensiveness and clinical applicability of ultrasound image three-dimensional reconstruction.

[0012] Further provided is that the S100 specifically comprises the following steps:

[0013] Acquiring multi-modal ultrasound image data; wherein the multi-modal ultrasound image data comprises ultrasound image sequences from different imaging modes, and each frame of image in the multi-modal ultrasound image data has an image acquisition timestamp and an imaging mode identifier;

[0014] Acquiring ultrasound image sequences from different imaging modes; wherein the different imaging modes include at least two of a B-mode ultrasound imaging mode, a Doppler ultrasound imaging mode and an elastic ultrasound imaging mode, and the ultrasound image sequences include ultrasound images acquired at different time points;

[0015] Extracting the image acquisition timestamp and the imaging mode identifier of each frame of image in the ultrasound image sequences; wherein the image acquisition timestamp is used to represent the absolute time of image acquisition, and the imaging mode identifier is used to uniquely identify the imaging mode to which the image belongs;

[0016] Performing image preprocessing on the ultrasound image sequences to obtain preprocessed ultrasound image sequences; wherein the image preprocessing includes at least one of image denoising, image enhancement and image size standardization;

[0017] constructing a multi-modal ultrasound image dataset based on the image acquisition timestamp, the imaging mode identifier and the pre-processed ultrasound image sequence; wherein the multi-modal ultrasound image dataset is an integrated ultrasound image collection from at least two different imaging modes associated with the timestamp and the imaging mode identifier.

[0018] By adopting the above technical solution, by specifically acquiring multi-modal ultrasound image data including ultrasound image sequences from at least two of different imaging modes such as B-mode ultrasound imaging mode, Doppler ultrasound imaging mode and elastic ultrasound imaging mode, and ensuring that each frame of image is provided with an image acquisition timestamp and an imaging mode identifier, time synchronization and mode recognition of multi-modal data can be realized; by performing image pre-processing on the ultrasound image sequence including image denoising, image enhancement and image size standardization, image quality is improved and noise interference is reduced, so that subsequent feature extraction is more reliable; based on the image acquisition timestamp, the imaging mode identifier and the pre-processed ultrasound image sequence, a multi-modal ultrasound image dataset is constructed to ensure the integrity and consistency of the dataset, laying a solid foundation for subsequent feature extraction and fusion, thereby improving the reliability and efficiency of the entire three-dimensional reconstruction process.

[0019] Further, in the S200, the generation process of the structure feature set includes:

[0020] based on the Canny edge detection algorithm, performing edge detection on each frame of the pre-processed ultrasound image to extract the anatomical structure contour information in the ultrasound image; based on the morphological opening operation and the morphological closing operation, performing boundary strengthening processing on the anatomical structure contour information to obtain the strengthened organ boundary information; based on the Harris corner point detection algorithm, performing corner point detection on each frame of the pre-processed ultrasound image to extract the key point position information in the ultrasound image; wherein the key point position information includes a pixel coordinate set of a plurality of key points and a corner point response intensity value corresponding to each key point, and the key point is an anatomical structure corner point or a tissue junction corner point in the ultrasound image;

[0021] integrating the anatomical structure contour information, the strengthened organ boundary information and the key point position information, and associating the corresponding image acquisition timestamp and the imaging mode identifier, to generate the structure feature set.

[0022] By adopting the technical scheme, the anatomic structure contour information in the ultrasound image is extracted based on the Canny edge detection algorithm, the boundary details of organs and tissues can be accurately captured, the boundary is strengthened based on the morphological opening operation and the morphological closing operation, the boundary information is further optimized to reduce the noise influence, and the definition of the structural features is improved; the key point position information including the pixel coordinates and the corner response intensity value is extracted based on the Harris corner detection algorithm, the anatomic structure corner or the tissue junction corner can be identified, the point accuracy of the features is enhanced, the anatomic structure contour information, the strengthened organ boundary information and the key point position information are integrated and associated with the time stamp and the imaging mode identifier, the structural feature set is generated, the integrity and the traceability of the structural features are ensured, reliable structural data basis is provided for subsequent registration and fusion, and therefore the structural accuracy and the detail retention capability of the three-dimensional reconstruction model are improved.

[0023] Further, in the S200, the generation process of the texture feature set includes:

[0024] Texture features of each frame of the preprocessed ultrasound image are extracted based on a gray level co-occurrence matrix algorithm, and a gray level co-occurrence matrix in different directions in the ultrasound image is obtained; a plurality of texture statistics are calculated based on the gray level co-occurrence matrix, and the texture statistics include at least three of contrast, correlation, energy and homogeneity; local texture features of each frame of the preprocessed ultrasound image are extracted based on a local binary pattern algorithm, and a local binary pattern coding sequence of each pixel point in the ultrasound image is obtained; each frame of the preprocessed ultrasound image is subjected to multi-scale and multi-directional filtering processing based on a Gabor filter bank, and texture response features of the ultrasound image in different frequencies and directions are extracted.

[0025] The texture statistics, the local binary pattern coding sequence and the texture response features are integrated, and corresponding image acquisition time stamps and imaging mode identifiers are associated, and the texture feature set is generated.

[0026] By adopting the technical scheme, texture features are extracted and texture statistics such as contrast, correlation, energy and homogeneity are calculated based on the gray level co-occurrence matrix algorithm, so that the texture properties in the ultrasound image can be quantified, and rich texture information can be provided for distinguishing different tissue types; local texture features are extracted based on the local binary pattern algorithm to obtain a local binary pattern coding sequence, so that subtle texture changes can be captured to enhance the local consistency of features; texture response features are extracted by performing multi-scale and multi-direction filtering based on a Gabor filter bank, so that texture details in different frequencies and directions can be covered, and the comprehensiveness and robustness of the texture features can be improved; the texture statistics, the local binary pattern coding sequence and the texture response features are integrated and associated with a time stamp and an imaging mode identifier to generate a texture feature set, so that the multidimensionality and integrability of the texture features can be ensured, detailed texture data can be provided for subsequent fusion, and thus the texture authenticity and tissue distinguishing capability of the three-dimensional reconstruction model can be enhanced.

[0027] Further, in the S200, the generation process of the function feature set includes:

[0028] The preprocessed ultrasound image sequence from the Doppler ultrasound imaging mode is subjected to spectral analysis based on a spectral Doppler analysis algorithm, and hemodynamic attribute information is extracted, the hemodynamic attribute information including blood flow velocity distribution information and blood flow direction information; wherein the blood flow velocity distribution information includes maximum flow velocity, average flow velocity and flow velocity profile, and the blood flow direction information is used to identify the direction of blood flow relative to the probe;

[0029] The preprocessed ultrasound image sequence from the elastic ultrasound imaging mode is subjected to elastic estimation based on an elastic imaging algorithm, and tissue mechanical attribute information is extracted, the tissue mechanical attribute information including tissue strain distribution information and elastic modulus distribution information; wherein the tissue strain distribution information is used to describe the deformation degree of the tissue under external pressure, and the elastic modulus distribution information is used to reflect the hardness attribute of the tissue;

[0030] The hemodynamic attribute information and the tissue mechanical attribute information are integrated and associated with corresponding image acquisition time stamps and imaging mode identifiers to generate the function feature set.

[0031] By adopting the technical scheme, the blood flow dynamics attribute information including blood flow velocity distribution information and blood flow direction information is extracted by performing spectrum analysis on the images of the Doppler ultrasound imaging mode based on a spectrum Doppler analysis algorithm, which can accurately reflect blood flow states such as maximum flow velocity, average flow velocity, flow velocity profile and blood flow direction, and provide key data for functional evaluation; the tissue mechanics attribute information including tissue strain distribution information and elastic modulus distribution information is extracted by performing elasticity estimation on the images of the elastic ultrasound imaging mode based on an elasticity imaging algorithm, which can quantize tissue hardness and deformation degree, and is helpful for identifying lesion tissues; the blood flow dynamics attribute information and the tissue mechanics attribute information are integrated and associated with time stamps and imaging mode identifiers; a functional feature set is generated, the real-time performance and mode specificity of the functional features are ensured, a functional dimension is injected into three-dimensional reconstruction, the reconstructed model can not only display structures but also simulate functional behaviors, and therefore the comprehensiveness and accuracy of diagnosis are improved.

[0032] Further, in the S300, the process of adding spatial coordinate information to the three-dimensional feature atlas includes:

[0033] The structure feature set, the texture feature set and the functional feature set are respectively subjected to time stamp alignment and imaging mode identifier alignment, to obtain a time-mode aligned multi-modal feature subset; the time-mode aligned multi-modal feature subset includes a structure feature subset, a texture feature subset and a functional feature subset under the same image acquisition time stamp and the same imaging mode identifier.

[0034] Based on the time-mode aligned multi-modal feature subset, the structure feature subset, the texture feature subset and the functional feature subset are respectively subjected to spatial coordinate registration, to obtain a spatially registered structure feature atlas, a spatially registered texture feature atlas and a spatially registered functional feature atlas; the spatial coordinate registration process includes rigid registration and non-rigid registration based on feature points; each feature point in the spatially registered structure feature atlas, the spatially registered texture feature atlas and the spatially registered functional feature atlas has three-dimensional coordinate information under a unified spatial coordinate system.

[0035] By adopting the technical scheme, the time-mode aligned multi-modal feature subset is obtained by time stamp alignment and imaging mode identification alignment of the structural feature set, the texture feature set and the functional feature set, the consistency of different modal features in time and mode is ensured, and errors caused by acquisition time or mode difference are eliminated; the spatial coordinate registration based on the aligned feature subset includes rigid registration and non-rigid registration based on feature points, the spatially registered structural feature atlas, the texture feature atlas and the functional feature atlas are obtained, each feature point is provided with unified three-dimensional coordinate information, accurate alignment of features in space is realized, the spatial correlation between features is enhanced, a consistent spatial framework is provided for subsequent fusion, and therefore the spatial accuracy of the three-dimensional reconstruction model and the feature integration effect are improved.

[0036] Further, in the S300, the process of the three-dimensional feature atlas with feature weights includes:

[0037] The spatially registered structural feature atlas, the texture feature atlas and the functional feature atlas are subjected to multi-modal feature fusion to generate a preliminary fusion feature atlas; wherein the multi-modal feature fusion includes weighted fusion based on feature weights and feature selection fusion based on an attention mechanism, and each feature point in the preliminary fusion feature atlas contains a structural feature component, a texture feature component and a functional feature component and a corresponding feature fusion weight.

[0038] By adopting the technical scheme, the preliminary fusion feature atlas is generated by multi-modal feature fusion including weighted fusion based on feature weights and feature selection fusion based on an attention mechanism on the spatially registered structural feature atlas, the texture feature atlas and the functional feature atlas, the weights can be allocated according to the importance of features, the feature contribution is optimized, the key features are highlighted, and the redundant or noise features are suppressed; each feature point in the preliminary fusion feature atlas contains a structural feature component, a texture feature component and a functional feature component and a corresponding feature fusion weight, the flexibility and adaptability of the fusion process are ensured, the fusion result can better reflect the actual clinical requirements, the comprehensiveness and representativeness of the three-dimensional reconstruction model are enhanced, and a foundation is laid for subsequent optimization.

[0039] Further, the S300 further includes the following steps:

[0040] The preliminary fusion feature atlas is subjected to spatio-temporal consistency optimization to obtain an optimized three-dimensional feature atlas; wherein the spatio-temporal consistency optimization includes adjacent time stamp feature smoothing processing and feature consistency constraint between imaging modes; the feature points in the optimized three-dimensional feature atlas have consistency in the time dimension and the space dimension;

[0041] Assign spatial coordinate information and feature weight to each feature point in the optimized three-dimensional feature map, and generate the fused three-dimensional feature map; wherein the spatial coordinate information is used to represent the position of the feature point in the three-dimensional space, and the feature weight is used to represent the importance degree of the feature point in the three-dimensional reconstruction process.

[0042] By adopting the above technical solution, the consistency of the feature points in the time dimension and the space dimension is ensured by performing the space-time consistency optimization on the preliminary fused feature map, including the feature smoothing processing based on the adjacent time stamps and the feature consistency constraint between the imaging modes, the inconsistency caused by the time sequence fluctuation and the mode difference is reduced, the stability and reliability of the features are improved, the spatial coordinate information and the feature weight are assigned to each feature point in the optimized three-dimensional feature map, the fused three-dimensional feature map is generated, the spatial position and the importance index of each feature point are ensured, the three-dimensional reconstruction process can be accurately modeled based on the weighted features, the space-time coherence and the feature accuracy of the final model are enhanced, and the overall reconstruction quality is improved.

[0043] Further, the S400 further includes the following steps:

[0044] Perform voxelization processing on the fused three-dimensional feature map to generate a three-dimensional voxel data set; wherein each voxel in the three-dimensional voxel data set contains a structural feature value, a texture feature value and a functional feature value mapped from the three-dimensional feature map, and each voxel has spatial coordinate information and feature weight;

[0045] Based on the feature weight of each voxel in the three-dimensional voxel data set, the structural feature value, the texture feature value and the functional feature value are weighted and fused to generate a set of voxel feature values after weighted fusion; wherein the set of voxel feature values after weighted fusion is used to represent the comprehensive feature intensity of each voxel in the three-dimensional space;

[0046] Based on the set of voxel feature values after weighted fusion, an isosurface is extracted by using a marching cubes algorithm to generate a preliminary three-dimensional mesh model; then, the preliminary three-dimensional mesh model is subjected to mesh optimization processing to obtain an optimized three-dimensional mesh model; wherein the mesh optimization processing includes at least two of mesh smoothing processing, mesh simplification processing and feature preservation processing, and the optimized three-dimensional mesh model reduces the number of meshes while retaining feature details;

[0047] Map and associate the optimized three-dimensional mesh model with the feature attribute information in the fused three-dimensional feature map to generate a three-dimensional reconstruction model of the ultrasound image.

[0048] By adopting the technical scheme, the three-dimensional voxel data set is generated by voxelizing the fused three-dimensional feature map, so that each voxel contains the structural feature value, the texture feature value and the functional feature value obtained by mapping, and has spatial coordinate information and feature weight, and the feature data is converted into a voxel format for facilitating three-dimensional processing; the voxel feature values are weighted and fused based on the feature weight to generate a set of weighted and fused voxel feature values, so as to realize comprehensive integration of the features and optimize the feature intensity representation at the voxel level; the moving cube algorithm is adopted based on the set of weighted and fused voxel feature values to extract an isosurface to generate a preliminary three-dimensional mesh model, so that the surface mesh can be efficiently constructed; the preliminary three-dimensional mesh model is subjected to mesh optimization processing including mesh smoothing processing, mesh simplification processing and feature preservation processing, so that the feature details are preserved while the number of meshes is reduced, and the rendering efficiency and the visual quality of the model are improved; finally, the optimized mesh model is mapped and associated with the feature attribute information to generate a three-dimensional reconstruction model of the ultrasound image, so that the model is not only geometrically accurate but also rich in feature attributes, and the practicability and the diagnostic value of the model are enhanced.

[0049] In summary, the present application has the following beneficial effects: through the synergistic fusion of multi-modal features and the spatio-temporal consistency optimization, a high-precision three-dimensional model that retains structural, texture and functional attributes at the same time can be generated, and the clinical applicability and the diagnostic value of the reconstruction result are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A flowchart of the embodiment is shown in the figure;

[0051] Figure 2 A flowchart of step S100 in the embodiment is shown in the figure;

[0052] Figure 3 A flowchart of step S200 in the embodiment is shown in the figure;

[0053] Figure 4 A flowchart of step S300 in the embodiment is shown in the figure;

[0054] Figure 5 A flowchart of step S400 in the embodiment is shown in the figure. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below with reference to the accompanying drawings.

[0056] As shown in the figure; Figures 1 to 5

[0057] The embodiment discloses an ultrasound image three-dimensional reconstruction method fusing multi-modal features, including the following steps:

[0058] ​S100. Acquire multimodal ultrasound image data; wherein, the multimodal ultrasound image data includes ultrasound image sequences from different imaging modes, and each frame of the multimodal ultrasound image data carries an image acquisition timestamp and an imaging mode identifier. Specifically, the steps include:

[0059] The system first acquires multimodal ultrasound image data, which includes ultrasound image sequences from different imaging modes, specifically covering image data from at least two of the following imaging modes: B-mode ultrasound, Doppler ultrasound, and elastography. Each image frame is accompanied by an image acquisition timestamp and an imaging mode identifier. The image acquisition timestamp precisely records the absolute time of image acquisition, while the imaging mode identifier uniquely identifies the imaging mode type to which the image belongs, thus providing the necessary time and mode correlation information for subsequent multimodal data alignment and fusion. After acquiring the raw image sequences, the system further performs image preprocessing operations on the ultrasound image sequences. This preprocessing may include at least one of the following methods: image denoising, image enhancement, and image size normalization, aiming to improve image quality, eliminate noise interference, and standardize image size to facilitate subsequent feature extraction and fusion processing. Image denoising can employ median filtering or wavelet denoising methods to suppress common speckle noise in ultrasound images; image enhancement can utilize histogram equalization or contrast-limited adaptive histogram equalization algorithms to strengthen structural details and texture information in the images; image size standardization adjusts all input images to a uniform resolution, eliminating scale inconsistencies caused by differences in acquisition equipment or parameters. After preprocessing, the system constructs a multimodal ultrasound image dataset based on image acquisition timestamps, imaging mode identifiers, and preprocessed ultrasound image sequences. This dataset is essentially a collection of ultrasound images from at least two different imaging modes, precisely correlated by timestamps and imaging mode identifiers, laying the data foundation for subsequent multimodal feature extraction, registration, and 3D reconstruction.

[0060] In practice, an ultrasound probe array equipped with a time synchronization module is used to acquire dynamic images at a sampling rate of 30 frames per second. Each frame of the image is embedded with an image acquisition timestamp accurate to milliseconds (generated by a high-precision clock chip), and the source is distinguished by a three-bit binary encoding of the imaging mode: for example, "001" represents B-mode, "010" represents Doppler mode, and "011" represents elastic mode. During the acquisition process, the system monitors the contact pressure between the probe and the tissue in real time. When the pressure value exceeds the safe range of 5-20N, a calibration procedure is automatically triggered to ensure the consistency of the physical parameters of the image. The original ultrasound image sequence undergoes layered preprocessing: First, a non-local mean filtering algorithm is used for image denoising, with a 7×7 pixel search window and a 3×3 pixel similarity window set, and a noise variance threshold set to 0.02, eliminating speckle noise while preserving the integrity of tissue boundaries. Then, contrast-limited adaptive histogram equalization is used for image enhancement, with a block size of 8×8 pixels and a contrast limit threshold of 2.0, significantly improving the separation between blood vessel walls and soft tissues in low-contrast areas. Finally, image size normalization was performed, resampling the resolution of different imaging modes to a uniform 512×512 pixels, calibrating the spatial resolution to 0.5 mm / pixel, and ensuring geometric feature integrity through bicubic interpolation. Next, a multimodal ultrasound image dataset was constructed, specifically using a four-dimensional tensor data structure (width × height × time frame × modal channel), aligned according to image acquisition timestamps. The alignment process employed a flexible matching algorithm based on dynamic time warping, with a maximum allowable time deviation of 33 ms (corresponding to a 30-frame sampling period). Imaging mode identifiers were used as metadata index keys to establish cross-modal association mappings within the dataset. The resulting structured dataset contains at least 2000 temporally distributed images, supporting rapid retrieval by timestamp range (e.g., cardiac cycle RR interval) or modal combination (e.g., B-mode + flexible synchronous view).

[0061] S200. Multimodal feature extraction is performed on the multimodal ultrasound image data to obtain a multimodal feature set; wherein, the multimodal feature set includes a structural feature set, a texture feature set, and a functional feature set, the structural feature set being used to describe anatomical structure information in the ultrasound image, the texture feature set being used to describe tissue texture information in the ultrasound image, and the functional feature set being used to describe hemodynamic and tissue mechanical property information in the ultrasound image. Specifically, the steps include:

[0062] In the generation process of the structural feature set, the system first performs edge detection on each pre-processed ultrasound image based on the Canny edge detection algorithm to extract the anatomical structure contour information in the image. Specifically, the Canny algorithm uses a double-threshold detection mechanism, with the high threshold set to 0.7 times the upper limit value of the gradient amplitude and the low threshold set to 0.4 times the high threshold, to effectively identify the tissue boundary and organ contour. Next, the system applies morphological opening and closing operations to the extracted contour information for boundary enhancement processing: the opening operation uses a 3x3 pixel circular structural element to eliminate small noise and artifacts; the closing operation also uses a 3x3 pixel structural element to fill in the contour interruptions and holes, and finally obtains the enhanced organ boundary information. In addition, the system performs corner detection on each image based on the Harris corner detection algorithm, sets the k value in the corner response function to 0.04, and the window size to 5x5 pixels, to extract the key point position information in the image, including the pixel coordinate set of multiple key points and the corresponding corner response intensity values. These key points mainly include anatomical structure corner points (such as vessel bifurcations, valve attachment points) or tissue junction corner points (such as soft tissue and bone interface). Finally, the system integrates the extracted anatomical structure contour information, enhanced organ boundary information and key point position information, and associates the corresponding image acquisition time stamp and imaging mode identifier to generate the structural feature set. The set is stored in the form of a feature vector, with each record containing a feature type identifier, spatial position, intensity value and time-modal metadata, providing a high-precision structural basis for subsequent multi-modal registration and fusion.

[0063] In the process of texture feature set generation, the system adopts multiple texture analysis algorithms to extract multi-level texture features from each pre-processed ultrasound image. First, based on the gray level co-occurrence matrix algorithm, the system calculates the gray level co-occurrence matrix of the image in four directions of 0°, 45°, 90° and 135°, with a step size of 1 pixel and a gray level quantization of 16 levels. Then, four texture statistics of contrast, correlation, energy and homogeneity are extracted from each matrix to form a set of 16-dimensional texture feature vectors. Second, the system applies the local binary pattern algorithm, using an 8-neighborhood and a circular neighborhood setting with a radius of 1 pixel, to encode the local texture of each pixel in the image and generate a local binary pattern encoding sequence. By calculating the LBP histogram of each 8x8 image block, a 59-dimensional LBP feature representation is obtained. Further, the system uses a Gabor filter bank for multi-scale and multi-directional filtering: four scales (wavelengths of 2, 4, 8, and 16 pixels) and six directions (0°, 30°, 60°, 90°, 120°, and 150°) are set, and a total of 24 Gabor filters are used for convolution operation. The mean and variance of each filter response map are extracted as texture response features, resulting in a 48-dimensional Gabor feature vector. Finally, the system integrates the above texture statistics, local binary pattern encoding sequence and Gabor texture response features, and associates them with the image acquisition timestamp and imaging mode identifier to generate the texture feature set. This set is organized in the form of a multi-dimensional feature tensor, supporting fast retrieval and comparison by time frame and modal type, and providing rich organizational texture representation for subsequent feature fusion.

[0064] In the generation process of the functional feature set, the system extracts hemodynamic and tissue mechanical attribute information for different imaging modes. For image sequences from Doppler ultrasound imaging mode, the system performs spectral analysis based on spectral Doppler analysis algorithm: first, the Doppler signal is subjected to fast Fourier transform, the window length is set to 256 points, and the overlap rate is 50%, the blood flow velocity spectrum of each time frame is extracted; further, the maximum flow velocity, average flow velocity and flow velocity profile (flow velocity distribution histogram) are calculated, and the blood flow direction (towards or away from the probe) is determined through phase information, forming hemodynamic attribute information. For image sequences from elastic ultrasound imaging mode, the system applies elastic imaging algorithm to extract tissue mechanical attribute: the cross-correlation algorithm is used to calculate the tissue displacement between consecutive frames, the displacement estimation window is set to 5x5 pixels, and the search range is ±10 pixels; further, the strain distribution is calculated according to the displacement field, the strain window is set to 3x3 pixels, and the strain is mapped to elastic modulus distribution through the Hook's law assumption, and the elastic modulus atlas reflecting tissue hardness is obtained. Finally, the system integrates the extracted hemodynamic attribute information (including blood flow velocity distribution and direction information) and tissue mechanical attribute information (including tissue strain distribution and elastic modulus distribution), and associates the image acquisition time stamp and imaging mode identifier, to generate a functional feature set. The set is stored in the form of space-time-functional four-dimensional data block, each functional feature point has a time stamp, modal identifier, spatial position and functional attribute value, which provides a data basis for subsequent construction of fused three-dimensional functional feature atlas.

[0065] S300, multi-modal feature registration and fusion is performed on the multi-modal feature set to obtain a fused three-dimensional feature atlas; wherein the fused three-dimensional feature atlas includes a spatially registered structural feature atlas, a texture feature atlas and a functional feature atlas, and each feature point in the fused three-dimensional feature atlas has spatial coordinate information and feature weight. Specifically, it includes the following steps:

[0066] Firstly, the structure feature set, the texture feature set and the function feature set are respectively time-stamped and imaging mode-identified. This process is achieved by establishing a unified time-modality index table, in which each feature entry is associated with its original image acquisition timestamp and imaging mode identification. The system uses an elastic matching algorithm based on dynamic time warping to align the feature sequences of Doppler and elasticity modes in the time dimension, with the time axis of the B-mode ultrasound image sequence as the reference. The maximum allowed time deviation is set to 33 milliseconds to ensure that different modal features collected at the same time can be accurately corresponded. The imaging mode identification serves as a modality differentiation key to ensure that features of different imaging modes at the same timestamp can be correctly classified and organized. After time-mode alignment, a time-mode aligned multi-modal feature subset is obtained, which contains the structure feature subset, the texture feature subset and the function feature subset under the same image acquisition timestamp and the same imaging mode identification. Then, based on the time-mode aligned multi-modal feature subset, the structure feature subset, the texture feature subset and the function feature subset are respectively spatially registered. The spatial coordinate registration process includes two stages of rigid registration and non-rigid registration. In the rigid registration stage, a feature point-based registration method is used, with the Harris corner points in the B-mode ultrasound image as reference points. The rotation matrix and translation vector between the feature points of Doppler and elasticity images and the reference points are calculated by the iterative closest point algorithm to achieve preliminary spatial alignment. In the non-rigid registration stage, thin plate spline transformation or B-spline transformation is used, with the organ boundary information in the structure features and the high response region in the texture features as control points, to optimize the preliminary registration result in fine granularity and eliminate local deformation errors caused by tissue deformation or probe movement. Finally, each feature point in the spatially registered structure feature atlas, texture feature atlas and function feature atlas is assigned a coordinate information in a unified three-dimensional coordinate system, with the ultrasound probe as the origin, the X and Y axes representing the lateral and longitudinal directions of the image respectively, and the Z axis representing the depth direction. The coordinate unit is millimeter.

[0067] Subsequently, the spatially registered structure feature atlas, texture feature atlas and function feature atlas are fused to generate a preliminary fusion feature atlas. Multi-modal feature fusion includes weighted fusion based on feature weights and feature selection fusion based on attention mechanism. In weighted fusion, the system assigns a dynamic weight coefficient to each feature dimension, which is adaptively adjusted according to the feature type (structure, texture or function) and the importance of the current imaging mode. For example, in the blood vessel region, the blood flow velocity information in the function feature has a higher weight; in the tissue boundary region, the edge intensity in the structure feature has a higher weight. The weighted formula is represented as ; wherein, , , represent the structure, texture and function feature components respectively. 、 、 are the weights of the structure, texture and function feature components, respectively, and satisfy ; represents the integrated feature intensity value after the fusion of structure, texture and function information at a spatial point. In the attention mechanism fusion, the system calculates the attention score of each feature point through a lightweight convolutional neural network, which is generated based on the local feature context and global semantic information, to enhance the representation of important feature regions and suppress redundant information. Finally, each feature point in the preliminary fusion feature map contains the structure feature component, the texture feature component and the function feature component, as well as the corresponding feature fusion weight.

[0068] Finally, the preliminary fusion feature map is optimized for spatio-temporal consistency to obtain the optimized three-dimensional feature map. The spatio-temporal consistency optimization includes feature smoothing based on adjacent timestamps and feature consistency constraints between different imaging modes. In the time dimension, the system uses Kalman filtering or Gaussian process regression to smooth the feature values at consecutive timestamps, reducing the temporal jitter caused by acquisition noise or motion artifacts. In the spatial dimension, the feature representation in different imaging modes is constrained by a multi-modal consistency loss function to ensure consistent feature responses at the same spatial position in different modalities. The feature points in the optimized three-dimensional feature map have high consistency in both time and space dimensions. The system then assigns spatial coordinate information and feature weights to each feature point in the optimized three-dimensional feature map to generate the final fused three-dimensional feature map. The spatial coordinate information is used to represent the position of the feature point in the three-dimensional space, and the feature weight is used to represent the importance degree of the feature point in the subsequent three-dimensional reconstruction process. The higher the feature weight, the greater the contribution of the feature point to the reconstruction result. The above process ensures the high unity and fusion of multi-modal features in the time, space and semantic dimensions, providing an accurate and robust feature basis for subsequent three-dimensional reconstruction.

[0069] S400, performing three-dimensional reconstruction processing on the fused three-dimensional feature map to generate a three-dimensional reconstruction model of the ultrasound image; wherein the three-dimensional reconstruction processing includes voxelization processing, feature weighted fusion, isosurface extraction and mesh optimization. Specifically, it includes the following steps:

[0070] The system first voxelizes the fused three-dimensional feature map to discretize it into a three-dimensional voxel data set. The voxelization process takes a voxel size of 0.5mm x 0.5mm x 0.5mm as the basic spatial unit to construct a three-dimensional grid covering the entire region of interest. Each voxel maps its corresponding structural feature value, texture feature value and functional feature value from the three-dimensional feature map through a trilinear interpolation algorithm, and inherits the spatial coordinate information and feature weight of the original feature point. The structural feature value includes edge strength and corner response strength, the texture feature value includes gray level co-occurrence matrix statistics, local binary pattern histogram vector and Gabor filter response value, and the functional feature value includes blood flow velocity value, elastic modulus value or strain value. Each voxel finally forms a multi-dimensional feature vector, the dimension of which is equal to the sum of the number of all feature types, and the feature weight value of the voxel is appended at the end of the vector, which is used to adjust the contribution of different features to the reconstruction result in the subsequent weighted fusion process. Then, the system weights and fuses the structural feature value, texture feature value and functional feature value based on the feature weight of each voxel in the three-dimensional voxel data set to generate a set of voxel feature values after weighted fusion. The weighted fusion process is ; wherein, represents the voxel located at the spatial coordinate The value of the integrated feature intensity of the voxel after weighted fusion; , and respectively represent the structural feature value, texture feature value and functional feature value of the voxel; , , respectively represent the structural feature value, texture feature value and functional feature value corresponding to the feature weight, and the value is derived from the feature weight of the position in the three-dimensional feature map. Through the above weighting process, the system can adaptively fuse multi-modal information according to the feature importance of different regions, for example, to enhance the weight of hemodynamic features in the blood vessel region, and to strengthen the representation of structural features in the tissue boundary region.

[0071] Subsequently, the system extracts the isosurface based on the weighted fused voxel feature value set using the marching cubes algorithm to generate a preliminary three-dimensional mesh model. The isosurface threshold of the marching cubes algorithm is dynamically set according to the imaging modality and clinical needs, and is usually set to the 60th percentile of the overall feature intensity value. The algorithm traverses each voxel unit, determines the topological structure of the isosurface in the unit according to the relative relationship between the feature intensity values of the eight vertices and the threshold, and calculates the accurate position of the isosurface vertex through linear interpolation. The final generated preliminary three-dimensional mesh model is represented in the form of a triangle patch set, including vertex coordinates, patch connection relationship and vertex normal vector information. In order to further improve the model quality, the system performs mesh optimization processing on the preliminary three-dimensional mesh model, which includes mesh smoothing processing, mesh simplification processing and feature preservation processing. The mesh smoothing processing uses the Laplace smoothing algorithm, the iteration number is set to 3 times, and the smoothing coefficient is 0.5, which is used to eliminate patch noise and jagged edges; the mesh simplification processing uses the edge folding algorithm, the target patch number is set to 30% of the original patch number, and in the simplification process, the Quadric Error Metric is used to keep the overall shape error of the model to be the minimum; the feature preservation processing identifies and preserves the key anatomical structure edges and high curvature areas through the feature perception constraint mechanism in the simplification and smoothing process, avoiding important features from being excessively smoothed or deleted in the optimization process. Finally, the system maps and associates the optimized three-dimensional mesh model with the feature attribute information in the fused three-dimensional feature map to generate a three-dimensional reconstruction model of the ultrasound image. The mapping process is realized through spatial coordinate matching, which maps each mesh vertex back to the nearest feature point in the original three-dimensional feature map, inherits its structure, texture and functional attributes, and is attached to the model data in the form of vertex attributes or texture maps. The final generated three-dimensional reconstruction model supports multi-modal feature visualization, and users can choose to display blood flow dynamics overlay, elastic modulus distribution or structure-texture fusion view, providing rich and intuitive multi-modal three-dimensional image support for clinical diagnosis.

[0072] The specific embodiments are only an explanation of the present application, which is not a limitation of the present application. Those skilled in the art can make modifications to the embodiments without creative contribution after reading the specification, as long as the modifications are within the scope of the claims of the present application.

Claims

1. An ultrasound image three-dimensional reconstruction method fusing multi-modal features, characterized in that, The method comprises the following steps: S100, acquiring multi-modal ultrasound image data; wherein the multi-modal ultrasound image data comprises ultrasound image sequences from different imaging modes, and each frame of image in the multi-modal ultrasound image data is provided with an image acquisition timestamp and an imaging mode identifier; S200, performing multi-modal feature extraction on the multi-modal ultrasound image data to obtain a multi-modal feature set; wherein the multi-modal feature set comprises a structural feature set, a texture feature set and a functional feature set, the structural feature set is used to describe anatomical structure information in the ultrasound image, the texture feature set is used to describe tissue texture information in the ultrasound image, and the functional feature set is used to describe blood flow hemodynamic and tissue mechanical property information in the ultrasound image; The generation process of the functional feature set comprises: performing spectral analysis on the preprocessed ultrasound image sequence from the Doppler ultrasound imaging mode based on a spectral Doppler analysis algorithm to extract blood flow hemodynamic property information, wherein the blood flow hemodynamic property information comprises blood flow velocity distribution information and blood flow direction information; wherein the blood flow velocity distribution information comprises maximum flow velocity, average flow velocity and flow velocity profile, and the blood flow direction information is used to identify the direction of blood flow relative to the probe; performing elasticity estimation on the preprocessed ultrasound image sequence from the elastic ultrasound imaging mode based on an elastic imaging algorithm to extract tissue mechanical property information, wherein the tissue mechanical property information comprises tissue strain distribution information and elastic modulus distribution information; wherein the tissue strain distribution information is used to describe the deformation degree of the tissue under external pressure, and the elastic modulus distribution information is used to reflect the hardness property of the tissue; integrating the blood flow hemodynamic property information and the tissue mechanical property information, and associating the corresponding image acquisition timestamp and imaging mode identifier to generate the functional feature set; S300, performing multi-modal feature registration and fusion on the multi-modal feature set to obtain a fused three-dimensional feature map; wherein the fused three-dimensional feature map comprises a spatially registered structural feature map, a texture feature map and a functional feature map, and each feature point in the fused three-dimensional feature map is provided with spatial coordinate information and feature weight; S400, performing three-dimensional reconstruction processing on the fused three-dimensional feature map to generate a three-dimensional reconstruction model of the ultrasound image; wherein the three-dimensional reconstruction processing comprises voxelization processing, feature weighted fusion, isosurface extraction and mesh optimization. 2.The method of claim 1, wherein, The S100 specifically comprises the following steps: acquiring multi-modal ultrasound image data; wherein the multi-modal ultrasound image data comprises ultrasound image sequences from different imaging modes, and each frame of image in the multi-modal ultrasound image data is provided with an image acquisition timestamp and an imaging mode identifier; acquiring ultrasound image sequences from different imaging modes; wherein the different imaging modes comprise at least two of a B-mode ultrasound imaging mode, a Doppler ultrasound imaging mode and an elastic ultrasound imaging mode, and the ultrasound image sequences comprise ultrasound images acquired at different time points; extracting an image acquisition timestamp and an imaging mode identifier of each frame of the sequence of ultrasound images; wherein the image acquisition timestamp is used to represent an absolute time of image acquisition, and the imaging mode identifier is used to uniquely identify an imaging mode to which the image belongs; performing image preprocessing on the sequence of ultrasound images to obtain a sequence of preprocessed ultrasound images; wherein the image preprocessing comprises at least one of image denoising, image enhancement, and image size standardization; constructing a multi-modal ultrasound image dataset based on the image acquisition timestamp, the imaging mode identifier, and the sequence of preprocessed ultrasound images; wherein the multi-modal ultrasound image dataset is a collection of ultrasound images from at least two different imaging modes that are associated by timestamps and imaging mode identifiers. 3.The method of claim 2, wherein, In the S200, the generation process of the structure feature set comprises: performing edge detection on each frame of the preprocessed ultrasound images based on a Canny edge detection algorithm to extract anatomical structure contour information in the ultrasound images; performing boundary enhancement processing on the anatomical structure contour information based on morphological opening and morphological closing operations to obtain enhanced organ boundary information; performing corner point detection on each frame of the preprocessed ultrasound images based on a Harris corner point detection algorithm to extract key point position information in the ultrasound images; wherein the key point position information comprises a pixel coordinate set of a plurality of key points and a corner point response intensity value corresponding to each key point, and the key points are anatomical structure corner points or tissue junction corner points in the ultrasound images; integrating the anatomical structure contour information, the enhanced organ boundary information, and the key point position information, and associating corresponding image acquisition timestamps and imaging mode identifiers to generate the structure feature set. 4.The method of claim 2, wherein, In the S200, the generation process of the texture feature set comprises: performing texture feature extraction on each frame of the preprocessed ultrasound images based on a gray level co-occurrence matrix algorithm to obtain gray level co-occurrence matrices in different directions in the ultrasound images; calculating a plurality of texture statistics based on the gray level co-occurrence matrices, wherein the texture statistics comprise at least three of contrast, correlation, energy, and homogeneity; performing local texture feature extraction on each frame of the preprocessed ultrasound images based on a local binary pattern algorithm to obtain a local binary pattern encoding sequence of each pixel point in the ultrasound images; performing multi-scale and multi-directional filtering processing on each frame of the preprocessed ultrasound images based on a Gabor filter bank to extract texture response features of the ultrasound images in different frequencies and directions; integrating the texture statistics, the local binary pattern encoding sequence, and the texture response features, and associating corresponding image acquisition timestamps and imaging mode identifiers to generate the texture feature set. 5.The method of claim 1, wherein, In the S300, the process of including spatial coordinate information in the three-dimensional feature map comprises: The structure feature set, the texture feature set and the function feature set are respectively subjected to timestamp alignment and imaging mode identification alignment to obtain a time-mode aligned multi-modal feature subset; wherein, the time-mode aligned multi-modal feature subset comprises a structure feature subset, a texture feature subset and a function feature subset under the same image acquisition timestamp and the same imaging mode identification; Based on the time-mode aligned multi-modal feature subset, the structure feature subset, the texture feature subset and the function feature subset are respectively subjected to spatial coordinate registration to obtain a spatially registered structure feature atlas, a texture feature atlas and a function feature atlas; wherein, the spatial coordinate registration process comprises rigid registration and non-rigid registration based on feature points; each feature point in the spatially registered structure feature atlas, texture feature atlas and function feature atlas is provided with three-dimensional coordinate information under a unified spatial coordinate system. 6.The method of claim 5, wherein, In the S300, the process of providing the three-dimensional feature atlas with feature weights comprises: The spatially registered structure feature atlas, texture feature atlas and function feature atlas are subjected to multi-modal feature fusion to generate a preliminary fusion feature atlas; wherein, the multi-modal feature fusion comprises weighted fusion based on feature weights and feature selection fusion based on an attention mechanism, and each feature point in the preliminary fusion feature atlas comprises a structure feature component, a texture feature component and a function feature component and corresponding feature fusion weights. 7.The method of claim 6, wherein, The S300 further comprises the following steps: The preliminary fusion feature atlas is subjected to spatio-temporal consistency optimization to obtain an optimized three-dimensional feature atlas; wherein, the spatio-temporal consistency optimization comprises adjacent timestamp feature smoothing processing and feature consistency constraint between imaging modes; the feature points in the optimized three-dimensional feature atlas have consistency in both time dimension and space dimension; Each feature point in the optimized three-dimensional feature atlas is assigned with spatial coordinate information and feature weights to generate the fused three-dimensional feature atlas; wherein, the spatial coordinate information is used to represent the position of the feature point in the three-dimensional space, and the feature weights are used to represent the importance degree of the feature point in the three-dimensional reconstruction process. 8.The method of claim 1, wherein, The S400 further comprises the following steps: The fused three-dimensional feature atlas is subjected to voxelization processing to generate a three-dimensional voxel data set; wherein, each voxel in the three-dimensional voxel data set comprises a structure feature value, a texture feature value and a function feature value mapped from the three-dimensional feature atlas, and each voxel is provided with spatial coordinate information and feature weights; Based on the feature weights of each voxel in the three-dimensional voxel data set, the structure feature value, the texture feature value and the function feature value are subjected to weighted fusion to generate a weighted fusion voxel feature value set; wherein, the weighted fusion voxel feature value set is used to represent the comprehensive feature intensity of each voxel in the three-dimensional space; Based on the weighted fused voxel feature value set, an iso-surface is extracted by using a marching cubes algorithm to generate a preliminary three-dimensional mesh model; then, the preliminary three-dimensional mesh model is subjected to mesh optimization processing to obtain an optimized three-dimensional mesh model; wherein the mesh optimization processing includes at least two of mesh smoothing processing, mesh simplification processing and feature preserving processing, and the optimized three-dimensional mesh model retains feature details while reducing the number of meshes; The optimized three-dimensional mesh model is mapped and associated with feature attribute information in the fused three-dimensional feature map to generate a three-dimensional reconstruction model of the ultrasound image.

Images