Automatic 3D Reconstruction System for Anesthesia Anatomy Based on Multimodal Medical Imaging

Abstract

This application discloses an automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging, belonging to the field of image analysis technology. The system includes a server used to match various preoperative ultrasound images based on rigid constraints of spatial anchor points in a unified coordinate system, determining a reference ultrasound image that best matches the physiological state of the static three-dimensional model; and to construct a dynamic three-dimensional model of the anesthesia anatomy based on the soft tissue deformation differences between the reference ultrasound image and various intraoperative ultrasound images. This application solves the technical problems in the prior art such as mismatch between static three-dimensional models and real-time intraoperative anatomical structures, low multimodal image registration accuracy, and inability to reflect real-time soft tissue deformation, achieving dynamic three-dimensional visualization reconstruction of anesthesia-related anatomical structures. The reconstructed model can accurately reflect the real-time changes in the patient's intraoperative anatomical structures.

Description

Technical Field

[0001] This application relates to the field of image analysis technology, specifically a three-dimensional automatic reconstruction system for anesthesia anatomy based on multimodal medical images. Background Technology

[0002] Throughout the entire clinical anesthesia procedure, both preoperative anatomical assessment and intraoperative real-time guidance heavily rely on medical imaging technology. Currently, standard clinical protocols typically employ computed tomography (CT) imaging preoperatively to assess the patient's anatomy and plan the anesthesia strategy; intraoperatively, real-time ultrasound imaging provides visual guidance for the puncture procedure. However, in practical clinical application, these techniques have insurmountable technical limitations and cannot meet the clinical demands for precise anesthesia.

[0003] On the one hand, preoperative computed tomography (CT) images can only obtain static anatomical information of the patient in a specific position. However, during the actual anesthesia procedure, the patient's respiratory movements, heartbeat, and positional adjustments, as well as the external forces exerted by ultrasound probe pressure and needle advancement, all cause dynamic deformation of the anesthetic anatomical structures, except for soft tissues. This leads to spatial displacement of key structures such as nerves and blood vessels. In this case, relying solely on preoperative static images for planning will result in a severe spatial misalignment between "preoperative image and intraoperative entity." This misalignment is particularly pronounced in obese patients or those with anatomical variations, easily leading to puncture path deviation, anesthesia failure, and even serious perioperative complications such as nerve damage, bleeding, and local anesthetic toxicity.

[0004] On the other hand, although intraoperative ultrasound can provide real-time imaging of the anatomical structures under anesthesia, it can only acquire two-dimensional tomographic images, resulting in limited imaging field of view and insufficient identification of overall anatomical structures. It cannot provide surgeons with a comprehensive three-dimensional understanding of the anatomical relationships among multiple structures in the target area, such as nerves, blood vessels, bones, and fascia. Operational precision is highly dependent on the surgeon's clinical experience and spatial imagination, posing a significant risk of human error. Furthermore, current technology cannot achieve effective spatial registration between preoperative high-precision static anatomical images obtained from computed tomography (CT) scans and dynamic intraoperative temporal ultrasound images. It is difficult to establish a unified coordinate system for multimodal images, cannot simultaneously consider the global information of static anatomical structures and the dynamic deformation characteristics of soft tissues during surgery, and cannot construct a dynamic three-dimensional model that can reflect real-time changes in the patient's intraoperative anatomical structures. These limitations constitute a core technical bottleneck restricting further improvements in the precision and safety of clinical anesthesia. Summary of the Invention

[0005] The purpose of this application is to provide an automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical images, in order to solve the technical problem that existing technologies cannot construct dynamic three-dimensional models of users' anesthesia anatomy.

[0006] To achieve the above objectives, this application provides the following technical solution: A three-dimensional automated reconstruction system for anesthesia anatomy based on multimodal medical imaging, comprising: The reader is used to acquire the user's medical imaging data information, which includes computed tomography images and multiple preoperative ultrasound images. The server is used to obtain a static three-dimensional model containing anesthesia anatomical structures based on the medical imaging data information. Furthermore, based on the static three-dimensional model, spatial anchor points are located for the bones in each preoperative ultrasound image to establish a unified coordinate system for multimodal images. Furthermore, based on the rigid constraints of each spatial anchor point in the unified coordinate system, each preoperative ultrasound image is matched to determine the benchmark ultrasound image that is closest to the physiological state of the static three-dimensional model. Furthermore, based on the differences in soft tissue deformation between the baseline ultrasound image and various intraoperative ultrasound images, a dynamic three-dimensional model of the anesthetic anatomy is constructed.

[0007] As a specific solution in the technical solution of this application, the server is also used to perform preprocessing operations on the computed tomography images and various preoperative ultrasound images, establish a unified pixel coordinate system for each image, and synchronously acquire the spatial pose information of the ultrasound probe corresponding to each preoperative ultrasound image. Furthermore, the preprocessed computed tomography images are automatically identified and segmented using a deep learning segmentation network to obtain the anatomical structure segmentation results of bones, blood vessels, nerves and fascia. Furthermore, based on the anatomical structure segmentation results, a static three-dimensional model containing the anesthetic anatomical structure is generated.

[0008] As a specific solution in the technical solution of this application, the server is further used to obtain a first similarity corresponding to each preoperative ultrasound image based on the static three-dimensional model and each preoperative ultrasound image; the first similarity is at least used to characterize the degree of similarity between the bone region in the preoperative ultrasound image and the corresponding bone projection region in the static three-dimensional model. Furthermore, based on each first similarity, the random sampling consensus algorithm is used to complete the skeletal region traversal matching and obtain the skeletal spatial anchor points between multimodal images; Furthermore, a unified coordinate system for the multimodal image is obtained based on the geometric center of the skeletal spatial anchor point.

[0009] As a specific solution in the technical solution of this application, the server is further configured to acquire a first ultrasound image based on each preoperative ultrasound image; the first ultrasound image is any ultrasound image among the preoperative ultrasound images for which a corresponding first similarity has not been acquired. Furthermore, based on the spatial pose information of the ultrasound probe corresponding to the first ultrasound image, a bone projection image is obtained from the static three-dimensional model; the projection direction of the bone projection image is perpendicular to the imaging plane of the first ultrasound image. Furthermore, based on the first ultrasound image and the bone projection image, a first similarity corresponding to the first ultrasound image is obtained.

[0010] As a specific solution in this application, the server is further configured to obtain a first edge contour curve and a second edge contour curve based on the first ultrasound image and the bone projection image; the first edge contour curve is the bone edge contour curve in the first ultrasound image; the second edge contour curve is the bone edge contour curve in the bone projection image. Furthermore, the Hausdorf distance is obtained based on the first edge contour curve and the second edge contour curve; Furthermore, the first similarity is obtained based on the Hausdorff distance; the first similarity is negatively correlated with the Hausdorff distance.

[0011] As a specific solution in this application, the server is further configured to obtain a first Hu moment vector and a second Hu moment vector based on the first edge contour curve and the second edge contour curve; the first Hu moment vector is the Hu moment vector corresponding to the first edge contour curve; the second Hu moment vector is the Hu moment vector corresponding to the second edge contour curve. Furthermore, based on the first Hu moment vector and the second Hu moment vector, a vector difference is obtained; the vector difference is equal to the absolute value of the difference between the first Hu moment vector and the second Hu moment vector. Furthermore, the first similarity is obtained based on the vector difference and the Hausdorff distance; the first similarity is negatively correlated with the vector difference.

[0012] As a specific solution in the technical solution of this application, the server is further used to obtain a second similarity that corresponds one-to-one with each preoperative ultrasound image based on the rigid constraints of each spatial anchor point in the unified coordinate system; the second similarity is at least used to characterize the degree of matching similarity between the soft tissue region of the anesthetic anatomical structure in the preoperative ultrasound image and the corresponding soft tissue region in the static three-dimensional model projected in the unified coordinate system. Additionally, the preoperative ultrasound images corresponding to the maximum value of each second similarity are selected and determined as the baseline ultrasound images.

[0013] As a specific solution in the technical solution of this application, the server is further configured to acquire a second ultrasound image based on each preoperative ultrasound image; the second ultrasound image is any ultrasound image among the preoperative ultrasound images for which a corresponding second similarity has not been acquired. Furthermore, based on the spatial pose information of the ultrasound probe corresponding to the second ultrasound image and the unified coordinate system, the soft tissue region of the anesthetic anatomical structure in the static three-dimensional model is projected onto the imaging plane of the second ultrasound image to obtain a soft tissue projection image. Furthermore, based on the soft tissue projection image and the second ultrasound image, a third edge contour curve and a fourth edge contour curve are obtained; the third edge contour curve is the soft tissue edge contour curve in the soft tissue projection image; and the fourth edge contour curve is the soft tissue edge contour curve in the second ultrasound image. Furthermore, a third similarity is obtained based on the third edge contour curve and the fourth edge contour curve; And, based on the third similarity, a second similarity of the second ultrasound image is obtained.

[0014] As a specific solution in this application, the server is further configured to obtain a fourth similarity based on the second ultrasound image; the fourth similarity is the first similarity corresponding to the second ultrasound image. Furthermore, based on the third similarity and the fourth similarity, a second similarity of the second ultrasound image is obtained.

[0015] As a specific solution in the technical solution of this application, the server is also used to extract biomechanical property parameters of soft tissue corresponding to the anesthetic anatomical structure from the static three-dimensional model. The biomechanical property parameters include at least soft tissue density, elastic modulus and Poisson's ratio. Furthermore, under the unified coordinate system, pixel-level registration is performed on the soft tissue regions in the reference ultrasound image and each intraoperative ultrasound image to calculate the deformation displacement vector field of the soft tissue between the reference ultrasound image and each intraoperative ultrasound image. Furthermore, using the rigid constraints of each spatial anchor point of the skeleton in the unified coordinate system as fixed boundary conditions, and combining the biomechanical property parameters and deformation displacement vector field of the soft tissue, a biomechanical constitutive model that conforms to the mechanical properties of human soft tissue is constructed. Furthermore, based on the temporal information of intraoperative ultrasound images, the biomechanical constitutive model is dynamically parameterized to obtain a dynamic three-dimensional model of the anesthetic anatomy.

[0016] Compared with the prior art, the beneficial effects of this application are: This application integrates high-resolution rigid structural information from preoperative CT images with real-time soft tissue dynamic information from intraoperative and preoperative ultrasound images. First, a high-precision static 3D model of the anesthesia anatomy is constructed based on the CT images. Then, a unified coordinate system for multimodal images is established through skeletal spatial anchor point positioning. A benchmark ultrasound image that best matches the physiological state of the static model is selected. Finally, based on the soft tissue deformation differences between the benchmark image and the intraoperative ultrasound image, a dynamic 3D model of the anesthesia anatomy is constructed using a biomechanical constitutive model. This application solves the technical problems of mismatch between static 3D models and real-time intraoperative anatomical structures, low multimodal image registration accuracy, and inability to reflect real-time soft tissue deformation in existing technologies. It achieves dynamic 3D visualization reconstruction of anesthesia-related anatomical structures. The reconstructed model accurately reflects the real-time changes in the patient's intraoperative anatomical structures, providing precise and real-time 3D navigation support for clinical anesthesia operations, effectively improving the safety and accuracy of anesthesia procedures. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the automatic three-dimensional reconstruction method for anesthesia anatomy based on multimodal medical images proposed in this application. Figure 2 This is a schematic diagram of the structure of the three-dimensional automatic reconstruction system for anesthesia anatomy based on multimodal medical images proposed in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. For example, the first ultrasound image and the second ultrasound image mentioned below are different ultrasound images. It should be understood that such names can be used interchangeably where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms. None of these are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0020] To address the technical problem mentioned in the background art that existing technologies cannot construct dynamic three-dimensional models of users' anesthesia anatomy, this application proposes an embodiment of an automatic three-dimensional reconstruction method for anesthesia anatomy based on multimodal medical imaging. For example... Figure 1 As shown, the method for automatic three-dimensional reconstruction of anesthesia anatomy based on multimodal medical images includes steps 100 to 500.

[0021] Step 100: Obtain the user's medical imaging data.

[0022] This step provides the foundational data source for subsequent three-dimensional reconstruction of anesthesia-associated anatomy. By simultaneously acquiring high-resolution rigid structural information from CT images and real-time soft tissue dynamic information from preoperative ultrasound images, it provides multi-dimensional anatomical data support for subsequent multimodal image fusion and dynamic model construction. In other words, in this embodiment, the medical imaging data includes computed tomography (CT) images and multiple preoperative ultrasound images.

[0023] In this embodiment, the user is a patient undergoing clinical anesthesia, and the anesthetic anatomy structure refers to the relevant anatomical region corresponding to the clinical anesthesia operation, including but not limited to the spinal canal region corresponding to spinal anesthesia, the peripheral nerve and blood vessel course region corresponding to nerve block, the upper airway anatomical region corresponding to airway management, and the deep soft tissue and adjacent bone region corresponding to local anesthesia, etc. This embodiment does not limit these.

[0024] In this embodiment, the computed tomography (CT) image is a thin-slice CT scan image covering the anatomical structure of anesthesia, completed by the user before surgery. It can be a spiral CT plain scan or enhanced image with a slice thickness of 0.625mm to 1.250mm. This image has the advantages of high resolution of bone and hard tissue imaging, clear anatomical structure boundaries, and accurate spatial positioning. It is the core benchmark data for constructing a static three-dimensional model of the anatomical structure of anesthesia.

[0025] In this embodiment, multiple preoperative ultrasound images are continuous multi-frame ultrasound images covering the anesthetic anatomy structure, acquired by the user through an ultrasound probe before the intraoperative anesthesia operation. Each preoperative ultrasound image corresponds to a unique time sequence number and acquisition timestamp, which can completely record the morphological changes of the anesthetic anatomy structure caused by the patient's breathing and body position changes, providing real-time soft tissue deformation data for the construction of dynamic three-dimensional models.

[0026] In this embodiment, the acquisition time period, number of acquisition frames, real-time acquisition time window, and sampling frame rate of medical images (i.e., CT images and ultrasound images) can all be limited to adapt to the entire process requirements of preoperative assessment and intraoperative real-time guidance in clinical anesthesia operations. Specific limitations can be as follows: CT image acquisition period: Thin-slice CT scans are limited to 1 to 3 days before the patient's anesthesia procedure. During the scan, the patient should maintain a basic body position that matches the intraoperative anesthesia procedure (e.g., lateral decubitus position for spinal anesthesia, supine / lateral decubitus position for peripheral nerve block) to avoid deviations in the spatial position of anatomical structures due to positional differences. This ensures that the anatomical structures in the static three-dimensional model obtained from preoperative CT images are consistent with the basic posture of the intraoperative anatomical structures.

[0027] Preoperative ultrasound image acquisition time and frame count: Preoperative ultrasound images were acquired 5 to 10 minutes after the patient entered the operating room and before the start of anesthesia, with the number of frames limited to 200 to 500. During the acquisition process, the ultrasound probe was kept fixed at the target area of ​​the anesthetic anatomy and the images were continuously acquired to fully cover 1 to 2 respiratory cycles plus heartbeat cycles of the patient. This ensured that the time-series images included the entire process of natural soft tissue deformation under the patient's physiological state, providing sufficient physiological state samples for the screening of benchmark ultrasound images.

[0028] Intraoperative ultrasound image acquisition time window and sampling frame rate: Intraoperative ultrasound images are image data acquired in real time during the anesthesia procedure. The acquisition time window is the entire anesthesia procedure (from puncture preparation to the end of the procedure, generally 5 to 30 minutes). The sampling frame rate of intraoperative ultrasound images can be consistent with the clinical routine scanning frame rate of ultrasound equipment, limited to 15 to 30 frames / second. This frame rate can accurately capture real-time deformation of soft tissues (e.g., organ movement caused by respiration, tissue deformation caused by probe pressure), while controlling the amount of data to avoid overloading the computing power of the server in real time and ensuring that the synchronous update of the dynamic three-dimensional model is delayed.

[0029] Acquisition supplementary rules for special scenarios: For special patients with obesity, anatomical variations, or respiratory / heart rate abnormalities, the number of frames acquired for preoperative ultrasound imaging can be increased to 500 to 800, and the acquisition time window can be extended to 2 to 3 respiratory / heartbeat cycles; the sampling frame rate for intraoperative ultrasound imaging can be adjusted to 25 to 30 frames per second according to clinical needs to ensure the accuracy of deformation capture, while the server adapts to dynamic scheduling of computing power to ensure the efficiency of model reconstruction.

[0030] In this embodiment, there are no restrictions on the source of medical imaging data. For example, CT images can be DICOM format CT image data retrieved from the hospital's image archiving and communication system; multiple preoperative ultrasound images can be DICOM format ultrasound image sequences acquired and transmitted in real time by anesthesia ultrasound equipment, or pre-stored ultrasound image sequences corresponding to anesthesia anatomical structures, or ultrasound image data with spatial pose information acquired synchronously by an intraoperative ultrasound probe.

[0031] It is important to note that in clinical anesthesia applications, users (e.g., anesthesiologists) need to monitor the dynamic changes in the patient's anesthetic anatomy in real time to avoid the risk of damaging blood vessels and nerves during anesthetic puncture. However, existing technologies can only view static anesthetic anatomy based on preoperative CT images, and cannot accurately integrate with intraoperative ultrasound images, making it difficult to intuitively reflect the real-time deformation of the anesthetic anatomy during surgery. Therefore, in one embodiment of this application, during preoperative ultrasound image acquisition, the spatial pose information of the ultrasound probe corresponding to each frame of preoperative ultrasound image can be simultaneously acquired using optical and electromagnetic positioning sensors mounted on the ultrasound probe. This spatial pose information includes the three-dimensional spatial coordinates of the ultrasound probe, the deflection angle of the imaging plane, the imaging depth, and the field of view, providing a pose reference for subsequent spatial registration of multimodal images. The relevant pose information is stored in association with the corresponding preoperative ultrasound image.

[0032] Step 200: Based on the medical imaging data, obtain a static three-dimensional model containing the anatomical structures under anesthesia.

[0033] The core of this step is to construct a high-precision static three-dimensional model of the anesthesia anatomy based on preoperative high-resolution CT images. This model clarifies the spatial location and morphological boundaries of key anesthesia-related anatomical structures such as bones, blood vessels, nerves, and fascia, providing a rigid benchmark for subsequent multimodal image registration and a basic three-dimensional topology for the construction of dynamic three-dimensional models. In other words, in this embodiment, any reasonable method can be used to obtain a static three-dimensional model containing the anesthesia anatomy based on the medical image data. For example, in one embodiment of this application, the method disclosed in patent application CN115713590A (titled "A CT-based Three-dimensional Reconstruction Image Processing Method and System") or patent application CN119722989A (titled "A CT Image Three-dimensional Reconstruction Method and Related Device Based on Morphological Gradient and Central Feature Extraction") can be used to obtain the static three-dimensional model containing the anesthesia anatomy.

[0034] In order to obtain a more accurate static three-dimensional model, in one embodiment of this application, step 200, based on the medical imaging data information, obtains a static three-dimensional model containing anesthesia anatomical structures, which may include steps 210 to 230.

[0035] Step 210: Perform preprocessing operations on the computed tomography images and each preoperative ultrasound image to establish a unified pixel coordinate system for each image and simultaneously acquire the spatial pose information of the ultrasound probe corresponding to each preoperative ultrasound image.

[0036] In this embodiment, the preprocessing operation is to eliminate noise, artifacts and other interference factors in the images (i.e., computed tomography images and various preoperative ultrasound images), unify the spatial specifications and pixel standards of the images, improve the accuracy of subsequent image segmentation, feature matching and spatial registration, and solve the technical problems of large segmentation errors and low registration accuracy caused by uneven image quality and inconsistent pixel standards in the prior art.

[0037] In this embodiment, the preprocessing operations for CT images include, but are not limited to, the following steps: Format conversion and data parsing: Analyze DICOM format CT images, extract key metadata such as pixel matrix, slice thickness, slice spacing, pixel spacing, scanning field of view, and patient position information, and convert image grayscale values ​​into Henle units to achieve standardized calibration of CT values; Noise and artifact suppression: Gaussian filtering, median filtering or anisotropic diffusion filtering algorithms are used to denoise CT images, eliminating Gaussian noise and salt-and-pepper noise generated during scanning. At the same time, a metal artifact correction algorithm is used to suppress metal artifacts caused by implants, dentures, etc., and improve image clarity. Region of Interest Extraction: Based on the anesthesia anatomical structure region input by the user, the CT image sequence is automatically cropped, retaining the effective image layer covering the anesthesia anatomical structure, and removing irrelevant image data to narrow the scope of subsequent data processing and improve processing efficiency; Gray-level normalization processing: The min-max normalization algorithm is used to map the HU value of CT images to the standard numerical range of [0,1], unifying the gray-level distribution range of the images and providing standardized input data for subsequent deep learning segmentation networks.

[0038] In this embodiment, the preprocessing operations for each preoperative ultrasound image include, but are not limited to, the following steps: Format conversion and data parsing: The preoperative ultrasound image sequence in DICOM format is parsed, and metadata such as pixel matrix, imaging depth, pixel spacing, acquisition timestamp, and probe model of each frame of ultrasound image is extracted. A unique time sequence number is assigned to each frame of image, and a one-to-one correspondence between the time sequence number and the image data is established. Noise suppression and image enhancement: A speckle suppression algorithm (e.g., SRAD algorithm or Lee filtering algorithm) is used to eliminate the speckle noise inherent in preoperative ultrasound images. At the same time, a contrast-limited adaptive histogram equalization algorithm is used to enhance the boundary contrast of soft tissues, blood vessels and nerves in preoperative ultrasound images and improve the recognizability of anatomical structures. Effective imaging area extraction: Automatically removes meaningless background areas, instrument parameter annotation areas, and timestamp display areas from preoperative ultrasound images, retaining only the effective scanning imaging area of ​​the ultrasound probe, thus avoiding interference from invalid areas with subsequent feature extraction and matching; Gray-scale normalization processing: The min-max normalization algorithm consistent with CT images is adopted to map the gray values ​​of preoperative ultrasound images to the standard numerical range of [0,1], so as to achieve the standardization of gray scale between CT images and preoperative ultrasound images, laying the foundation for subsequent multimodal image matching.

[0039] In this embodiment, a unified pixel coordinate system is established for each image. Specifically, the top left corner of the first frame of the CT image sequence is taken as the origin, the horizontal pixel direction of the image is taken as the X-axis, the vertical pixel direction is taken as the Y-axis, and the stacking direction of the CT image is taken as the Z-axis to establish a three-dimensional pixel coordinate system. For each frame of preoperative ultrasound image, based on its pixel spacing and imaging depth parameters, its two-dimensional pixel coordinate system is mapped to the reference plane of the above-mentioned three-dimensional pixel coordinate system to achieve the unification of pixel coordinate system for all images and ensure the consistency of coordinate reference in the subsequent spatial registration process.

[0040] In this embodiment, the spatial pose information of the ultrasound probe corresponding to each preoperative ultrasound image is acquired synchronously. Specifically, if the probe pose information has been acquired synchronously by the positioning sensor during the acquisition of the preoperative ultrasound image, the spatial pose information associated with each frame of the preoperative ultrasound image is directly extracted. If the pose information has not been acquired in advance, the visual odometry method based on the image sequence is used to calculate the relative pose change of the ultrasound probe corresponding to each frame of the preoperative ultrasound image by matching the feature points between adjacent preoperative ultrasound images, thereby obtaining the spatial pose information of the ultrasound probe corresponding to each frame of the preoperative ultrasound image.

[0041] It should be noted that the use of visual odometry to obtain the spatial pose information of the ultrasonic probe is a mature technology. For example, similar technologies are disclosed in patent documents such as CN115615427B, entitled "An Ultrasonic Probe Navigation Method, Device, Equipment and Medium", and CN111655156B, entitled "Combined Image-Based and Inertial Probe Tracking". This application can directly use existing mature algorithms to obtain the spatial pose information of the ultrasonic probe, which will not be elaborated further.

[0042] Step 220: Automatically identify and segment the anatomical structures of anesthesia in the preprocessed computed tomography images using a deep learning segmentation network to obtain the anatomical structure segmentation results of bones, blood vessels, nerves and fascia.

[0043] The core function of this step is to automatically and accurately identify and segment key anesthesia-related anatomical structures from CT images using deep learning networks, providing accurate structural boundary data for the construction of static 3D models, and solving the technical problems of low efficiency, poor accuracy, and insufficient consistency of segmentation results among different operators in existing technologies.

[0044] In this embodiment, the deep learning segmentation network is a pre-trained dedicated deep learning network for segmenting anesthesia-related anatomical structures. The backbone network of this network can adopt mature network architectures in the field of medical image segmentation such as U-Net, V-Net, ResU-Net, and Swin-UNETR. This embodiment does not limit this.

[0045] It should be noted that the use of deep learning networks for automatic identification and segmentation of anatomical structures in medical CT images is a mature technology. For example, similar technologies are disclosed in patent documents such as: CN110738660B, entitled "Spine CT Image Segmentation Method and Apparatus Based on Improved U-net"; CN114419083B, entitled "ResUnet Medical Image Segmentation System Based on Improved Edge Operator"; and CN113935976B, entitled "Automatic Segmentation Method and System for Intra-organ Blood Vessels in Enhanced CT Images".

[0046] In this embodiment, the training process of the deep learning segmentation network is as follows: A dataset of CT images of anesthesia anatomy structures is pre-constructed. This dataset contains CT images of anesthesia anatomy structures covering patients of different ages, genders, and body types. All images are manually annotated by senior anesthesiologists and radiologists, with anatomical structures such as bones, blood vessels, nerves, and fascia manually labeled to form gold standard segmentation labels. The dataset is divided into training, validation, and test sets in an 8:1:1 ratio. The deep learning segmentation network is then trained under supervision. A combination of Dice loss and cross-entropy loss functions is used to optimize the network parameters until the network achieves a Dice coefficient ≥ 0.92 on the test set, completing the network training and optimization. The trained deep learning segmentation network can automatically output binary segmentation masks corresponding to bones, blood vessels, nerves, and fascia in each layer of the input preprocessed CT image sequence, forming a complete anatomical structure segmentation result.

[0047] In this embodiment, the anatomical structure segmentation results include: skeletal structure segmentation masks, vascular structure segmentation masks, neural structure segmentation masks, and fascial structure segmentation masks. Each segmentation mask corresponds one-to-one with the pixel coordinate system of the CT image, enabling precise characterization of the boundaries and distribution range of each anatomical structure in three-dimensional space. Among them, the skeletal structure is a rigid structure, serving as a reference for spatial anchor point positioning in subsequent multimodal image registration; blood vessels, nerves, and fascia are soft tissue structures, which are core structures that need to be avoided or targeted during anesthesia operations, and are also the core objects of subsequent dynamic deformation monitoring.

[0048] In one embodiment of this application, to further improve segmentation accuracy, after the deep learning segmentation network outputs the initial segmentation results, morphological processing operations can be used to post-process the segmentation results, including but not limited to: using opening operations to eliminate isolated noise points in the segmentation results, using closing operations to fill small holes in the segmentation mask, and using connected component analysis to remove irrelevant connected components with an area smaller than a preset threshold, ultimately obtaining a segmentation result with smooth edges and a complete anatomical structure. For example, the preset threshold can be set to 0.05% of the total number of pixels in a single frame of the segmented image, retaining only connected components with an area larger than this threshold, further improving the accuracy of the segmentation results.

[0049] Step 230: Based on the anatomical structure segmentation results, generate a static three-dimensional model containing the anesthesia anatomical structure.

[0050] The core function of this step is to reconstruct a three-dimensional visualization model of the anesthetic anatomy based on the two-dimensional CT image segmentation results, clarify the three-dimensional spatial adjacency relationship between each anatomical structure, and provide a three-dimensional reference structure for subsequent multimodal image registration and dynamic model construction.

[0051] In this embodiment, the generation of the static 3D model can employ 3D reconstruction algorithms that use surface rendering or volume rendering, including but not limited to mature algorithms in the field of medical image 3D reconstruction such as the moving cube algorithm, the traveling tetrahedron algorithm, and the ray casting algorithm. This embodiment does not limit the specific algorithms used in this regard.

[0052] In one embodiment of this application, step 230, generating a static three-dimensional model containing the anesthesia anatomical structure based on the anatomical structure segmentation results, may include steps 231 to 233.

[0053] Step 231: For each type of anatomical structure segmentation result of bones, blood vessels, nerves, and fascia, the moving cube algorithm is used to reconstruct the three-dimensional surface of the segmentation mask of continuous multi-layer CT images to generate a three-dimensional mesh model corresponding to each type of anatomical structure. The three-dimensional mesh model is composed of vertices, edges, and faces, which can accurately represent the three-dimensional shape and boundary of the anatomical structure.

[0054] Step 232: Smooth and simplify the generated 3D mesh models of various anatomical structures. Use the Laplacian mesh smoothing algorithm to eliminate jagged edges in the mesh, and use the quadratic error measure algorithm to simplify the mesh. Without changing the overall shape of the model and the boundaries of key structures, reduce the number of vertices and faces of the mesh, thereby reducing the computational power consumption of subsequent model registration and deformation calculation.

[0055] Step 233: Spatially integrate the optimized 3D mesh models of various anatomical structures according to a unified 3D pixel coordinate system, configure corresponding visualization attributes such as color and transparency for different types of anatomical structure models, and generate static 3D models containing complete anatomical structures of bones, blood vessels, nerves and fascia.

[0056] In this embodiment, the generated static 3D model can be exported as a common 3D model format such as STL, OBJ, and PLY, supporting visualization, rotation, scaling, and sectioning operations in the anesthesia navigation system. At the same time, the model retains the 3D spatial coordinate information of each anatomical structure, providing a rigid benchmark for the spatial registration of subsequent multimodal images.

[0057] Step 300: Based on the static three-dimensional model, spatial anchor point positioning is performed on the bones in each preoperative ultrasound image to establish a unified coordinate system for multimodal images.

[0058] The core of this step is to accurately locate the spatial anchor points of the bones in preoperative ultrasound images based on the rigid skeletal structure in the static 3D model, and establish a unified spatial coordinate system between CT images and preoperative ultrasound images. This solves the technical problems of inconsistent coordinate systems in multimodal images, large registration errors of rigid structures, and insufficient accuracy of soft tissue fusion in existing technologies, providing a unified spatial reference for subsequent dynamic matching and deformation calculation of soft tissues. Specifically, step 300 involves locating spatial anchor points of the bones in each preoperative ultrasound image based on the static 3D model, establishing a unified coordinate system for multimodal images, including steps 310 to 330.

[0059] Step 310: Based on the static three-dimensional model and each preoperative ultrasound image, obtain the first similarity corresponding to each preoperative ultrasound image.

[0060] In this embodiment, the first similarity is used at least to characterize the degree of similarity between the skeletal region in the preoperative ultrasound image and the corresponding skeletal projection region in the static three-dimensional model.

[0061] The core of this step is to quantify the morphological similarity between the skeletal regions in each frame of preoperative ultrasound images and the corresponding skeletal projection regions in the static 3D model. This provides a quantitative basis for the subsequent selection and matching of skeletal spatial anchor points, ensuring that the selected spatial anchor points have sufficient morphological matching and reducing registration errors. Specifically, step 310, based on the static 3D model and each preoperative ultrasound image, obtains the first similarity score corresponding to each preoperative ultrasound image, including steps 311 to 313.

[0062] Step 311: Based on each preoperative ultrasound image, obtain the first ultrasound image.

[0063] In this embodiment, the first ultrasound image is any preoperative ultrasound image from among all preoperative ultrasound images for which a corresponding first similarity has not been obtained. That is, the purpose of this step is to traverse all preoperative ultrasound images and calculate the first similarity for each frame sequentially, ensuring that all preoperative ultrasound images can complete the bone morphology matching assessment with the static three-dimensional model. In other words, in this embodiment, the calculation method for the first similarity corresponding to any preoperative ultrasound image can refer to the first ultrasound image, and will not be elaborated further.

[0064] Step 312: Based on the spatial pose information of the ultrasound probe corresponding to the first ultrasound image, obtain the bone projection image from the static three-dimensional model.

[0065] In this embodiment, the projection direction of the obtained bone projection image is perpendicular to the imaging plane of the first ultrasound image.

[0066] The core of this step is to use the real-time pose information of the ultrasound probe to project the three-dimensional skeletal structure in the static three-dimensional model into a two-dimensional image from the same imaging angle as the preoperative ultrasound image, thereby generating a skeletal projection image that is completely matched with the imaging angle of the first ultrasound image, ensuring the consistency of the benchmark for subsequent morphological matching.

[0067] In this embodiment, the specific acquisition process of the bone projection image is as follows: First, based on the spatial pose information of the ultrasound probe corresponding to the first ultrasound image, the spatial position and normal vector direction of the ultrasound imaging plane in the three-dimensional coordinate system of the static three-dimensional model are determined; then, with the normal vector direction as the projection direction (i.e., the projection direction is perpendicular to the ultrasound imaging plane), the three-dimensional mesh model of the skeleton in the static three-dimensional model is orthogonally projected to generate a two-dimensional bone projection image that is completely consistent with the pixel size and imaging field of view of the first ultrasound image; the bone projection image is a binary image, in which the pixel value of the bone projection area is 1 and the pixel value of the background area is 0, which can clearly characterize the edge contour shape of the skeleton from the corresponding viewpoint.

[0068] In one embodiment of this application, the projection range can be limited according to the imaging depth parameters of the ultrasound probe, retaining only the bone structure projection within the ultrasound imaging depth range and eliminating bone structures outside the imaging field of view, ensuring that the generated bone projection image completely matches the bone imaging range in the first ultrasound image, and further improving the accuracy of subsequent similarity calculation.

[0069] Step 313: Based on the first ultrasound image and the bone projection image, obtain the first similarity corresponding to the first ultrasound image.

[0070] The core of this step is to extract the bone edge contour features from the first ultrasound image and the bone projection image, quantify the degree of morphological similarity between the two, and obtain a first similarity that can accurately characterize the matching degree of the bone region, providing a quantitative basis for the subsequent selection of spatial anchor points. In other words, in this embodiment, any reasonable method can be used to obtain the first similarity corresponding to the first ultrasound image based on the first ultrasound image and the bone projection image. In the field of image analysis, obtaining the similarity between two images (i.e., the first ultrasound image and the bone projection image) is a mature technology, and will not be elaborated upon here.

[0071] In order to accurately quantify the morphological matching degree of the bone region in the first ultrasound image and the corresponding bone projection image, effectively reduce the interference of noise and imaging field deviation of the first ultrasound image on the similarity calculation results, and provide a stable and reliable quantitative judgment benchmark for subsequent selection of bone spatial anchor points and rigid registration of multimodal images, in one embodiment of this application, step 313: based on the first ultrasound image and the bone projection image, obtain the first similarity corresponding to the first ultrasound image, including steps 313a to 313c.

[0072] Step 313a: Based on the first ultrasound image and the bone projection image, obtain the first edge contour curve and the second edge contour curve.

[0073] In this embodiment, the first edge contour curve is the bone edge contour curve in the first ultrasound image. The second edge contour curve is the bone edge contour curve in the bone projection image.

[0074] In this embodiment, the extraction process of the first edge contour curve can be as follows: First, a threshold segmentation algorithm is used to segment the skeletal region of the first ultrasound image. Since the skeleton appears as a hyperechoic bright area in the first ultrasound image, accompanied by acoustic shadows, a grayscale threshold of 0.7 (after normalization) can be set, and the area with a grayscale value greater than or equal to this threshold is initially determined as a candidate skeleton region. Then, morphological processing and connected component analysis are used to remove pseudo-hyperechoic areas caused by blood vessels, fascia, etc. in the first ultrasound image, and to retain connected components that conform to the morphological characteristics of the skeleton. Finally, the Canny edge detection algorithm is used to extract the edges of the segmented skeleton region to obtain a single-pixel-width, continuously closed first edge contour curve.

[0075] In this embodiment, the extraction process of the second edge contour curve can be as follows: for the binarized bone projection image, the Canny edge detection algorithm is directly used to extract the boundary of the bone projection area to obtain a second edge contour curve with the same specifications as the first edge contour curve, a single pixel width, and continuous closure.

[0076] It should be noted that the Canny edge detection algorithm for extracting structural edge contours in medical images is a mature technology, and will not be elaborated here.

[0077] Step 313b: Obtain the Hausdorff distance based on the first edge contour curve and the second edge contour curve.

[0078] In this embodiment, the Hausdorff distance is used to characterize the maximum mismatch between two point sets. Specifically, it is the larger of the maximum value of the shortest distance from all points on the first edge contour curve to the point set on the second edge contour curve, and the maximum value of the shortest distance from all points on the second edge contour curve to the point set on the first edge contour curve. The smaller the Hausdorff distance, the higher the morphological matching degree of the two edge contour curves; the larger the value, the greater the morphological difference between the two contour curves.

[0079] It should be noted that calculating the Hausdorff distance between two contour curves is a mature technique in the field of graphics processing, and will not be elaborated here. In this embodiment, the calculated Hausdorff distance is in pixels, which can be converted into the actual physical distance (unit: mm) based on the pixel spacing of the image, facilitating subsequent similarity measurement.

[0080] Step 313c: Obtain the first similarity based on the Hausdorff distance.

[0081] In this embodiment, the first similarity is negatively correlated with the Hausdorff distance. That is, in this embodiment, any reasonable method can be used to obtain the first similarity based on the Hausdorff distance, as long as the first similarity is negatively correlated with the Hausdorff distance. For example, the reciprocal of the Hausdorff distance can be used as the first similarity; or, in step 313c, the formula for calculating the first similarity based on the Hausdorff distance can be as follows:

[0082] in, Indicates the first similarity; This represents an exponential function with the natural constant e as its base. The preset adjustment coefficient can be set according to clinical registration accuracy requirements, for example: It can be set to 5; Indicates Hausdorff distance; This represents the total length of the second edge contour curve. This formula maps the Hausdorff distance to a numerical range of [0, 1], and the larger the Hausdorff distance, the smaller the first similarity value, satisfying a negative correlation. For example, in a specific embodiment, the calculated Hausdorff distance... The total length of the second edge contour curve is 2 pixels. For 200 pixels, Setting it to 5 will result in the first similarity score. =exp(-5×2 / 200)=exp(-0.05)≈0.951, indicating that the matching degree of the two contour curves (i.e. the first edge contour curve and the second edge contour curve) is extremely high.

[0083] It should be noted that calculating the first similarity based on Hausdorff distance can only represent the maximum degree of mismatch between two contour curves and cannot fully reflect the overall morphological characteristics of the contour curves. This can lead to situations where there are large differences in local contours but the overall shape matches, resulting in deviations in the similarity calculation results. To address this technical problem, in a preferred embodiment of this application, step 313c, obtaining the first similarity based on the Hausdorff distance, may further include steps 313e to 313g.

[0084] Step 313e: Based on the first edge contour curve and the second edge contour curve, obtain the first Hu moment vector and the second Hu moment vector.

[0085] In this embodiment, the first Hu moment vector is the Hu moment vector corresponding to the first edge contour curve. The second Hu moment vector is the Hu moment vector corresponding to the second edge contour curve.

[0086] It is important to understand that Hu moments are invariant to translation, rotation, and scaling, and can accurately represent the overall geometric features of two-dimensional graphics. They are unaffected by changes in the position, angle, or size of the contour curve, and are suitable for morphological matching of skeletal contours in multimodal images.

[0087] In this embodiment, the process of obtaining the first Hu moment vector and the second Hu moment vector is as follows: For the binarized images corresponding to the first edge contour curve and the second edge contour curve, calculate their 0th to 3rd order geometric moments respectively. Based on the geometric moments, calculate 7 invariant Hu moments, and form a 7-dimensional Hu moment vector from these 7 Hu moments, that is, obtain the first Hu moment vector = [h11, h12, ..., h17] and the second Hu moment vector = [h21, h22, ..., h27] respectively. It should be noted that calculating the Hu moment vector of a certain edge contour curve (i.e., the first edge contour curve and the second edge contour curve) is a mature technology in the field of digital image processing, and will not be elaborated here.

[0088] Step 313f: Obtain the vector difference based on the first Hu moment vector and the second Hu moment vector.

[0089] In this embodiment, the vector difference is equal to the absolute value of the difference between the first Hu moment vector and the second Hu moment vector.

[0090] In this embodiment, to eliminate the difference in numerical magnitude between different Hu moment components, the absolute logarithm of each component of the first and second Hu moment vectors can be taken first, and then the absolute value of the difference between the corresponding components can be calculated, ultimately obtaining a 7-dimensional vector difference = [|lg|h11|-lg|h21||,|lg|h12|-lg|h22||,...,|lg|h17|-lg|h27||]. The smaller the value of the vector difference, the closer the overall morphological characteristics of the two contour curves are.

[0091] Step 313g: Obtain the first similarity based on the vector difference and the Hausdorff distance.

[0092] In this embodiment, the first similarity is negatively correlated with the vector difference. That is, in this embodiment, any reasonable method can be used to obtain the first similarity based on the vector difference, as long as the first similarity is negatively correlated with the vector difference. For example, in one embodiment of this application, the first similarity can be the product or sum of the reciprocal of the vector difference and the reciprocal of the Hausdorff distance. In another embodiment of this application, step 313g, the formula for calculating the first similarity based on the vector difference and the Hausdorff distance, can be as follows:

[0093] in, Indicates the first similarity; Indicates the first weighting coefficient; This represents an exponential function with the natural constant e as its base. The preset adjustment coefficient can be set according to clinical registration accuracy requirements, for example: It can be set to 5; Indicates Hausdorff distance; This indicates the total length of the second edge contour curve; This represents the second weighting coefficient; m is the adjustment coefficient for the difference in Hu moment vectors, which can be set according to requirements, for example, it can be set to 2; The L2 norm represents the vector difference.

[0094] In this embodiment, the first weighting coefficient Second weighting coefficient The values ​​can be set according to registration requirements. For example, in a general scenario, α=0.6 and β=0.4 can be set. The first similarity calculated in this embodiment is still in the numerical range of [0,1], and the first similarity is negatively correlated with Hausdorff distance and vector difference, which can comprehensively characterize the local and global matching degree of the skeletal contour. For example, in a specific embodiment, α=0.6, β=0.4, k=5, m=2, the calculated value is... , =0.1, then The final first similarity score was 0.6×0.951+0.4×0.819≈0.5706+0.3276≈0.898, indicating a good overall matching degree for the skeletal region.

[0095] Step 320: Based on each first similarity, combine the random sampling consensus algorithm to complete the skeletal region traversal matching and obtain the skeletal spatial anchor points between multimodal images.

[0096] The core of this step is to select bone feature points with sufficiently high matching degree based on the first similarity corresponding to each frame of preoperative ultrasound image, and then eliminate mismatched points by combining random sampling consensus algorithm, so as to finally obtain high-precision and high-robust bone space anchor points, providing reference points for the establishment of a unified coordinate system.

[0097] In this embodiment, the random sampling consensus algorithm is a mature algorithm that can robustly estimate model parameters from a dataset containing a large number of outliers (i.e., mismatched points). It can effectively remove mismatched skeletal feature points caused by artifacts and noise in ultrasound images, ensuring that the selected spatial anchor points are all correctly matched inliers. Using the random sampling consensus algorithm for feature point matching and mismatch removal is a mature technology in the field of computer vision, and will not be elaborated here.

[0098] In one embodiment of this application, step 320 may include steps 321 to 324.

[0099] Step 321: Set a first similarity threshold and filter out all preoperative ultrasound images with a first similarity greater than or equal to the first similarity threshold as valid matching images; the first similarity threshold can be set according to the registration accuracy requirements, for example, it can be set to 0.75, only retaining images with a skeletal region matching degree that meet the standard, and removing invalid images with too low a matching degree. Step 322: For each frame of valid matching image, extract multiple feature points uniformly on the first edge contour curve, and at the same time extract the corresponding matching feature points on the second edge contour curve, establish a one-to-one correspondence between the feature points of the two-dimensional ultrasound image and the vertices of the skeleton in the three-dimensional static model, and form an initial set of matching point pairs.

[0100] Step 323: Using the initial set of matching point pairs as input, the random sampling consensus algorithm is used to iteratively solve the transformation matrix based on the three-dimensional rigid transformation model. In each iteration, the minimum number of matching point pairs are randomly selected to solve the transformation matrix. The number of interior points that conform to the transformation matrix is ​​counted. After a preset number of iterations (e.g., 1000 times), the optimal transformation matrix with the largest number of interior points is obtained. At the same time, all exterior points (mismatched point pairs) that do not conform to the optimal transformation matrix are removed.

[0101] Step 324: Use the 3D skeleton vertices corresponding to all the retained interior points after filtering as the skeleton space anchor points between multimodal images.

[0102] In this embodiment, the number of skeleton space anchor points obtained is no less than 3, and all space anchor points are not collinear, which can uniquely determine the rigid transformation relationship in three-dimensional space and provide sufficient reference constraints for the establishment of a unified coordinate system.

[0103] Step 330: Based on the geometric center of the skeleton spatial anchor point, obtain the unified coordinate system of the multimodal image.

[0104] The core of this step is to complete the registration and unification of the three-dimensional pixel coordinate system of CT images and the spatial coordinate system of preoperative ultrasound images based on the high-precision bone spatial anchor points obtained through screening, establish a shared spatial coordinate system for multimodal images, and solve the technical problem of inconsistent spatial references for multimodal images in existing technologies.

[0105] In one embodiment of this application, step 330 may include steps 331 to 333.

[0106] Step 331: Calculate the geometric center of the three-dimensional coordinates of all skeletal spatial anchor points, and use this geometric center as the origin of the coordinate system.

[0107] Step 332: Using the long axis of the bone obtained by fitting the bone spatial anchor points as the X-axis, the front-back direction perpendicular to the long axis of the bone as the Y-axis, and the up-down direction perpendicular to the XY plane as the Z-axis, establish a three-dimensional spatial coordinate system based on the right-hand rule. This coordinate system is the unified coordinate system for multimodal images.

[0108] Step 333: Based on the optimal transformation matrix obtained in step 323, the three-dimensional pixel coordinate system of the CT image and the two-dimensional pixel coordinate system of each frame of preoperative ultrasound image are transformed into the above unified coordinate system to complete the spatial coordinate unification of all image data.

[0109] In this embodiment, the established unified coordinate system ensures that all anatomical structures in the static three-dimensional model are in the same spatial reference as the imaging structures in each frame of preoperative ultrasound images. This provides a unified spatial coordinate system for subsequent soft tissue matching, deformation calculation, and dynamic model construction, fundamentally solving the spatial misalignment problem of multimodal image fusion.

[0110] Step 400: Based on the rigid constraints of each spatial anchor point in the unified coordinate system, match each preoperative ultrasound image and determine the benchmark ultrasound image that is closest to the physiological state of the static three-dimensional model.

[0111] The core of this step is to quantify the soft tissue matching degree between preoperative ultrasound images and static 3D models under the rigid constraints of a unified coordinate system, and screen out the benchmark ultrasound image that is closest to the patient's physiological state (such as respiratory phase and body position) during the preoperative CT scan. This provides a benchmark reference for subsequent soft tissue deformation calculations, solving the technical problem of mismatch between the physiological state of the static model and the intraoperative ultrasound image, and large deviation of the deformation calculation benchmark in the existing technology. Specifically, step 400, based on the rigid constraints of each spatial anchor point in the unified coordinate system, matches each preoperative ultrasound image to determine the benchmark ultrasound image that is closest to the physiological state of the static 3D model, and may include steps 410 and 420.

[0112] Step 410: Based on the rigid constraints of each spatial anchor point in the unified coordinate system, obtain the second similarity that corresponds one-to-one with each preoperative ultrasound image.

[0113] In this embodiment, the second similarity is at least used to characterize the degree of similarity between the soft tissue region of the anesthetic anatomy structure in the preoperative ultrasound image and the corresponding soft tissue region in the static three-dimensional model projected in a unified coordinate system.

[0114] In this embodiment, the calculation method for the second similarity can refer to the calculation method for the first similarity in step 310 above, and will not be repeated here.

[0115] In a specific embodiment of this application, step 410, based on the rigid constraints of each spatial anchor point in the unified coordinate system, obtains a second similarity that corresponds one-to-one with each preoperative ultrasound image, including steps 411 to 415.

[0116] Step 411: Based on each preoperative ultrasound image, acquire a second ultrasound image.

[0117] In this embodiment, the second ultrasound image is any ultrasound image among the various preoperative ultrasound images that did not obtain a corresponding second similarity.

[0118] Step 412: Based on the spatial pose information of the ultrasound probe corresponding to the second ultrasound image and the unified coordinate system, project the soft tissue region of the anesthetic anatomical structure in the static three-dimensional model onto the imaging plane of the second ultrasound image to obtain a soft tissue projection image.

[0119] The core of this step is to project the three-dimensional soft tissue structures such as blood vessels, nerves, and fascia in the static three-dimensional model into a two-dimensional image based on the pose information of the ultrasound probe under the reference of a unified coordinate system, according to the imaging perspective consistent with the second ultrasound image, to generate a soft tissue projection image that is completely matched with the imaging perspective and spatial position of the ultrasound image, so as to ensure the consistency of the reference for subsequent soft tissue matching.

[0120] In this embodiment, the method for obtaining soft tissue projection images can refer to the method for obtaining bone projection images in step 312 above, and will not be repeated here.

[0121] In one embodiment of this application, the spatial position of the projected image can be calibrated based on the rigid constraints of the bone spatial anchor points to ensure that the bone position in the soft tissue projection image completely coincides with the bone position in the second ultrasound image, thereby eliminating spatial offset during the projection process and further improving the accuracy of subsequent soft tissue contour matching.

[0122] Step 413: Based on the soft tissue projection image and the second ultrasound image, obtain the third edge contour curve and the fourth edge contour curve.

[0123] In this embodiment, the third edge contour curve is the soft tissue edge contour curve in the soft tissue projection image. The fourth edge contour curve is the soft tissue edge contour curve in the second ultrasound image.

[0124] In this embodiment, the method for obtaining the third edge contour curve and the fourth edge contour curve can refer to the method for obtaining the first edge contour curve and the second edge contour curve in step 313a above, and will not be repeated here.

[0125] Step 414: Obtain the third similarity based on the third edge contour curve and the fourth edge contour curve.

[0126] As mentioned above, obtaining the similarity between two contour curves (i.e., the third edge contour curve and the fourth edge contour curve) is a mature technology, and will not be elaborated here. For example, the calculation method of the third similarity can refer to the calculation method of the first similarity in step 313, that is, to calculate by fusing Hausdorff distance and Hu moment features. The final third similarity is in the numerical range of [0,1]. The closer the value is to 1, the higher the contour matching degree of the corresponding soft tissue.

[0127] In one embodiment of this application, if there are multiple types of target soft tissues (e.g., blood vessels, nerves, and fascia), the similarity of each type of soft tissue can be calculated separately. Then, based on the importance of each type of soft tissue in the anesthesia procedure, corresponding weight coefficients are configured, and a comprehensive third similarity is obtained by weighted summation. For example, for nerve block anesthesia scenarios, the weight coefficient of nerve structures can be set to 0.5, the weight coefficient of vascular structures can be set to 0.3, and the weight coefficient of fascia structures can be set to 0.2. The sum of all weight coefficients is 1, ensuring that the calculated third similarity can meet the actual needs of clinical anesthesia.

[0128] Step 415: Based on the third similarity, obtain the second similarity of the second ultrasound image.

[0129] In this embodiment, the obtained third similarity can be directly used as the second similarity of the second ultrasound image. It is important to understand that using the third similarity of soft tissue as the second similarity ignores the matching constraints of the rigid skeletal structure. This may result in situations where the soft tissue contour matches but the bone position is offset, causing the calculated second similarity to fail to accurately reflect the overall matching degree between the image and the model. Therefore, in one embodiment of this application, step 415, obtaining the second similarity of the second ultrasound image based on the third similarity, includes steps 415a and 415b.

[0130] Step 415a: Based on the second ultrasound image, obtain the fourth similarity.

[0131] In this embodiment, the fourth similarity is the first similarity corresponding to the second ultrasound image (that is, the similarity between the bone region in the second ultrasound image used to characterize its bone and the corresponding bone projection region in the static three-dimensional model in step 310).

[0132] Step 415b: Based on the third similarity and the fourth similarity, obtain the second similarity of the second ultrasound image.

[0133] In this embodiment, the second similarity of the second ultrasound image can be obtained based on the third similarity and the fourth similarity using any reasonable method. For example, the second similarity can be the average of the third and fourth similarities. Alternatively, the formula for calculating the second similarity can be as follows:

[0134] in, Indicates the second similarity; This represents the third weighting coefficient; Indicates third similarity; This represents the fourth weighting coefficient; The fourth similarity coefficient represents the third weighting coefficient. and the fourth weighting coefficient The sum is 1, and the third weighting coefficient and the fourth weighting coefficient The settings can be adjusted according to the clinical scenario. For example, for spinal anesthesia scenarios where the skeleton is used as the positioning reference, γ=0.6 and δ=0.4 can be set; for peripheral nerve block scenarios where soft tissue is the core, γ=0.4 and δ=0.6 can be set.

[0135] In a specific embodiment, γ=0.5, δ=0.5, the fourth similarity is 0.90, the third similarity is 0.88, then the second similarity... =0.5×0.90+0.5×0.88=0.89, indicating that the overall matching degree between the second ultrasound image and the static three-dimensional model is good.

[0136] Step 420: Select the preoperative ultrasound images corresponding to the maximum value of each second similarity and determine them as the baseline ultrasound images.

[0137] In this embodiment, the larger the value of the second similarity corresponding to a certain preoperative ultrasound image, the higher the overall matching degree between the preoperative ultrasound image and the static three-dimensional model, and the closer the corresponding patient's physiological state (breathing phase, body position, muscle tension, etc.) is to the physiological state during the preoperative CT scan.

[0138] In one embodiment of this application, a minimum threshold for the second similarity can be set, for example, to 0.75. If the second similarity of all preoperative ultrasound images is lower than the minimum threshold, a prompt message is issued to prompt the user to re-acquire preoperative ultrasound images of the anesthesia anatomy structure to ensure that the selected benchmark ultrasound images have sufficient matching accuracy and to avoid excessive errors in subsequent dynamic model reconstruction due to low benchmark image matching.

[0139] Step 500: Based on the differences in soft tissue deformation between the baseline ultrasound image and various intraoperative ultrasound images, construct a dynamic three-dimensional model of the anesthetic anatomy.

[0140] In this embodiment, intraoperative ultrasound imaging refers to a continuous sequence of ultrasound images that are acquired in real time by an ultrasound probe and cover the anatomical structures under anesthesia throughout the entire anesthesia procedure.

[0141] The core of this step is to construct a three-dimensional model that reflects the dynamic changes in the patient's intraoperative anatomical structure in real time, based on the differences in soft tissue deformation between the baseline ultrasound image and the intraoperative ultrasound image, combined with the biomechanical properties of soft tissue. This solves the technical problems of existing technologies being unable to construct dynamic three-dimensional models and the mismatch between static models and real-time intraoperative anatomical structures, providing precise real-time three-dimensional navigation for clinical anesthesia operations. Specifically, step 500 involves constructing a dynamic three-dimensional model of the anesthesia anatomy based on the differences in soft tissue deformation between the baseline ultrasound image and each intraoperative ultrasound image, including steps 510 to 540.

[0142] Step 510: Extract the biomechanical property parameters of the soft tissue corresponding to the anesthetic anatomical structure from the static three-dimensional model. The biomechanical property parameters include at least soft tissue density, elastic modulus and Poisson's ratio.

[0143] In this embodiment, the soft tissue includes anesthesia-related soft tissue structures such as blood vessels, nerves, fascia, and muscles. Different types of soft tissue have different biomechanical property parameters. In this embodiment, a human soft tissue biomechanical property parameter library can be pre-constructed. This parameter library can be built based on authoritative medical literature and clinical biomechanical research data, and includes standard parameters such as density, elastic modulus, and Poisson's ratio of various soft tissues corresponding to different ages, genders, and body mass indexes (BMI). For example, in a specific embodiment, for a 30-year-old male user with a BMI of 22, the matched muscle tissue elastic modulus is 12 kPa, Poisson's ratio is 0.49, and density is 1060 kg / m³; the fascia tissue elastic modulus is 50 kPa, Poisson's ratio is 0.48, and density is 1120 kg / m³; the blood vessel tissue elastic modulus is 200 kPa, Poisson's ratio is 0.45, and density is 1050 kg / m³. These parameters are assigned to the corresponding soft tissue models in the static three-dimensional model.

[0144] It should be noted that the values ​​of biomechanical property parameters of human soft tissue have been verified in a large number of clinical biomechanical studies. Relevant parameters can be found in publicly available medical and biomechanical literature, and will not be elaborated here.

[0145] Step 520: Under the unified coordinate system, perform pixel-level registration of the soft tissue regions in the reference ultrasound image and each intraoperative ultrasound image, and calculate the deformation displacement vector field of the soft tissue between the reference ultrasound image and each intraoperative ultrasound image.

[0146] The core of this step is to accurately calculate the deformation and direction of soft tissue under both baseline and real-time conditions, obtaining pixel-level deformation displacement vector fields, which provides accurate displacement boundary conditions for the deformation-driven process of subsequent biomechanical models.

[0147] In this embodiment, the intraoperative ultrasound images are ultrasound images of the corresponding anesthetic anatomical structures acquired in real time by the ultrasound probe during the anesthesia operation. Each frame of the intraoperative ultrasound image carries temporal information and corresponding spatial pose information of the ultrasound probe, and has been mapped to a unified coordinate system.

[0148] In this embodiment, a non-rigid registration algorithm based on the Demons algorithm is used to complete pixel-level registration between the reference ultrasound image and the intraoperative ultrasound image, and to calculate the deformation displacement vector field of the soft tissue. The Demons algorithm is a mature algorithm in the field of non-rigid registration of medical images. It can accurately calculate the local deformation between two images and obtain a pixel-level displacement vector field. The displacement vector corresponding to each pixel represents the deformation direction and deformation amount of the soft tissue at that location from the reference state to the real-time state.

[0149] It should be noted that the use of the Demons algorithm for non-rigid registration and deformation displacement field calculation of medical images is a mature technology. For example, similar technologies are disclosed in patent documents such as: CN110473234B, entitled "Differential Homeomorphism Demons Image Registration Method, System and Storage Medium"; and CN109978784B, entitled "MR Image and CT Image Registration Method, Device, Computer Equipment and Storage Medium".

[0150] In one embodiment of this application, in order to improve the accuracy of deformation calculation, a rigid constraint of the bone space anchor point can be added during the non-rigid registration process. That is, the pixel displacement vector of the bone region is fixed to 0, and only the deformation displacement of the soft tissue region is calculated. This ensures that the rigid structure of the bone does not deform, conforms to the actual movement law of human anatomy, and avoids the problem of bone and soft tissue deformation misalignment.

[0151] Step 530: Using the rigid constraints of each spatial anchor point of the skeleton in the unified coordinate system as fixed boundary conditions, and combining the biomechanical property parameters and deformation displacement vector field of the soft tissue, construct a biomechanical constitutive model that conforms to the mechanical properties of human soft tissue.

[0152] The core of this step is to construct a biomechanical constitutive model that can accurately simulate the deformation law of soft tissue based on the material properties and actual deformation displacement of soft tissue, and to extend the deformation information in two-dimensional ultrasound images to three-dimensional space, providing a mechanical driving model for the construction of dynamic three-dimensional models.

[0153] In this embodiment, the biomechanical constitutive model can be constructed using a finite element analysis model, specifically including steps 531 to 535.

[0154] Step 531: Perform tetrahedral mesh generation on the soft tissue 3D mesh model in the static 3D model to generate the finite element mesh of the soft tissue. At the same time, perform mesh generation on the skeletal structure as a rigid constraint body.

[0155] Step 532: Assign the soft tissue biomechanical property parameters extracted in Step 510 to the corresponding elements in the finite element mesh, and define the material constitutive model of the soft tissue. For soft tissues such as blood vessels and fascia that are approximately linearly elastic, a linear elastic constitutive model is used; for hyperelastic soft tissues such as muscles, a Neo-Hookean hyperelastic constitutive model is used to ensure that the model conforms to the actual mechanical properties of different soft tissues.

[0156] Step 533: Set each spatial anchor point of the skeleton in the unified coordinate system to a fixed boundary condition, that is, the node displacement of the skeleton mesh is fixed to 0, and no deformation or displacement occurs, which conforms to the rigidity characteristics of the human skeleton.

[0157] Step 534: Map the deformation displacement vector field calculated in step 520 onto the corresponding nodes of the soft tissue finite element mesh as the displacement boundary condition of the model, driving the soft tissue mesh to undergo displacement that conforms to the actual deformation law.

[0158] Step 535: Based on the finite element method, solve the equilibrium differential equation of the soft tissue to obtain the displacement and stress distribution of all nodes of the three-dimensional mesh of the soft tissue, and construct a biomechanical constitutive model that conforms to the mechanical properties of human soft tissue.

[0159] It should be noted that finite element modeling and biomechanical simulation of human soft tissue are mature technologies. For example, similar technologies are disclosed in patent documents such as: CN113470165B, entitled "A Soft Tissue Modeling Method Based on Radial Base Point Interpolation and Mass Spring Method"; and CN114241156B, entitled "A Device and Simulation System for Simulating Soft Tissue Deformation".

[0160] Step 540: Based on the temporal information of intraoperative ultrasound images, perform dynamic parameter calibration on the biomechanical constitutive model to obtain a dynamic three-dimensional model of the anesthetic anatomy.

[0161] In this embodiment, the generated dynamic three-dimensional model can be updated synchronously with the intraoperative ultrasound images. The update frame rate is consistent with the acquisition frame rate of the intraoperative ultrasound images (usually 15 to 30 frames / second). It can reflect the real-time deformation of the patient's anesthetic anatomy structure caused by breathing, body position changes, and tissue traction without delay. It provides accurate real-time three-dimensional navigation for clinical anesthesia puncture, nerve block and other operations, effectively avoiding the risk of intraoperative vascular and nerve damage.

[0162] In this application, after the construction of the dynamic 3D model is completed, the output method of the dynamic 3D model can be flexibly configured according to the application scenario of anesthesia navigation. For example, the dynamic 3D model can be output to the display terminal of the anesthesia navigation system in real time, simultaneously displaying the 3D visualization of the model, the real-time position of key anatomical structures, and the safe range of the puncture path; or the temporal deformation data of the dynamic 3D model can be stored synchronously to form a record of the dynamic changes in anatomical structures during the anesthesia operation, which can be used for postoperative review and teaching training.

[0163] The embodiment of the automatic three-dimensional reconstruction method for anesthesia anatomy based on multimodal medical imaging proposed in this application integrates high-resolution rigid structural information from preoperative CT images with real-time soft tissue dynamic information from intraoperative and preoperative ultrasound images. First, a high-precision static three-dimensional model of the anesthesia anatomy is constructed based on the CT images. Then, a unified coordinate system for the multimodal images is established through skeletal spatial anchor point positioning. A reference ultrasound image that best matches the physiological state of the static model is selected. Finally, based on the soft tissue deformation differences between the reference image and the intraoperative ultrasound image, a dynamic three-dimensional model of the anesthesia anatomy is constructed using a biomechanical constitutive model. This method solves the technical problems of mismatch between the static three-dimensional model and the real-time intraoperative anatomical structure, low multimodal image registration accuracy, and inability to reflect real-time soft tissue deformation in existing technologies. It achieves dynamic three-dimensional visualization reconstruction of anesthesia-related anatomical structures. The reconstructed model can accurately reflect the real-time changes in the patient's intraoperative anatomical structure, providing precise and real-time three-dimensional navigation support for clinical anesthesia operations, effectively improving the safety and accuracy of anesthesia procedures.

[0164] Having introduced the embodiments of the automatic three-dimensional reconstruction method for anesthesia anatomy based on multimodal medical images proposed in this application, the embodiments of the automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical images proposed in this application are described below. Figure 2 As shown, the three-dimensional automatic reconstruction system 10 for anesthesia anatomy based on multimodal medical images includes: Reader 11 is used to acquire the user's medical imaging data information; the medical imaging data information includes computed tomography images and multiple preoperative ultrasound images; Server 12 is used to obtain a static three-dimensional model containing anesthesia anatomical structures based on the medical imaging data information. Furthermore, based on the static three-dimensional model, spatial anchor points are located for the bones in each preoperative ultrasound image to establish a unified coordinate system for multimodal images. Furthermore, based on the rigid constraints of each spatial anchor point in the unified coordinate system, each preoperative ultrasound image is matched to determine the benchmark ultrasound image that is closest to the physiological state of the static three-dimensional model. Furthermore, based on the differences in soft tissue deformation between the baseline ultrasound image and various intraoperative ultrasound images, a dynamic three-dimensional model of the anesthetic anatomy is constructed.

[0165] As a specific embodiment of this application, the server 12 is further configured to perform preprocessing operations on the computed tomography images and various preoperative ultrasound images, establish a unified pixel coordinate system for each image, and synchronously acquire the spatial pose information of the ultrasound probe corresponding to each preoperative ultrasound image. Furthermore, the preprocessed computed tomography images are automatically identified and segmented using a deep learning segmentation network to obtain the anatomical segmentation results of bones, blood vessels, nerves and fascia. Furthermore, based on the anatomical structure segmentation results, a static three-dimensional model containing the anesthetic anatomical structure is generated.

[0166] As a specific embodiment of this application, the server 12 is further configured to obtain a first similarity corresponding to each preoperative ultrasound image based on the static three-dimensional model and each preoperative ultrasound image; the first similarity is at least used to characterize the degree of similarity between the bone region in the preoperative ultrasound image and the corresponding bone projection region in the static three-dimensional model. Furthermore, based on each first similarity, the random sampling consensus algorithm is used to complete the skeletal region traversal matching and obtain the skeletal spatial anchor points between multimodal images; Furthermore, a unified coordinate system for the multimodal image is obtained based on the geometric center of the skeletal spatial anchor point.

[0167] As a specific embodiment of this application, the server 12 is further configured to acquire a first ultrasound image based on each preoperative ultrasound image; the first ultrasound image is any ultrasound image among the preoperative ultrasound images for which a corresponding first similarity has not been acquired. Furthermore, based on the spatial pose information of the ultrasound probe corresponding to the first ultrasound image, a bone projection image is obtained from the static three-dimensional model; the projection direction of the bone projection image is perpendicular to the imaging plane of the first ultrasound image. Furthermore, based on the first ultrasound image and the bone projection image, a first similarity corresponding to the first ultrasound image is obtained.

[0168] As a specific embodiment of this application, the server 12 is further configured to obtain a first edge contour curve and a second edge contour curve based on the first ultrasound image and the bone projection image; the first edge contour curve is the bone edge contour curve in the first ultrasound image; the second edge contour curve is the bone edge contour curve in the bone projection image. Furthermore, the Hausdorf distance is obtained based on the first edge contour curve and the second edge contour curve; Furthermore, the first similarity is obtained based on the Hausdorff distance; the first similarity is negatively correlated with the Hausdorff distance.

[0169] As a specific embodiment of this application, the server 12 is further configured to obtain a first Hu moment vector and a second Hu moment vector based on the first edge contour curve and the second edge contour curve; the first Hu moment vector is the Hu moment vector corresponding to the first edge contour curve; the second Hu moment vector is the Hu moment vector corresponding to the second edge contour curve. Furthermore, based on the first Hu moment vector and the second Hu moment vector, a vector difference is obtained; the vector difference is equal to the absolute value of the difference between the first Hu moment vector and the second Hu moment vector. Furthermore, the first similarity is obtained based on the vector difference and the Hausdorff distance; the first similarity is negatively correlated with the vector difference.

[0170] As a specific embodiment of this application, the server 12 is further configured to obtain a second similarity corresponding to each preoperative ultrasound image based on the rigid constraints of each spatial anchor point in the unified coordinate system; the second similarity is at least used to characterize the degree of similarity between the soft tissue region of the anesthetic anatomical structure in the preoperative ultrasound image and the corresponding soft tissue region in the static three-dimensional model projected in the unified coordinate system. Additionally, the preoperative ultrasound images corresponding to the maximum value of each second similarity are selected and determined as the baseline ultrasound images.

[0171] As a specific embodiment of this application, the server 12 is further configured to acquire a second ultrasound image based on each preoperative ultrasound image; the second ultrasound image is any ultrasound image among the preoperative ultrasound images for which a corresponding second similarity has not been acquired. Furthermore, based on the spatial pose information of the ultrasound probe corresponding to the second ultrasound image and the unified coordinate system, the soft tissue region of the anesthetic anatomical structure in the static three-dimensional model is projected onto the imaging plane of the second ultrasound image to obtain a soft tissue projection image. Furthermore, based on the soft tissue projection image and the second ultrasound image, a third edge contour curve and a fourth edge contour curve are obtained; the third edge contour curve is the soft tissue edge contour curve in the soft tissue projection image; and the fourth edge contour curve is the soft tissue edge contour curve in the second ultrasound image. Furthermore, a third similarity is obtained based on the third edge contour curve and the fourth edge contour curve; And, based on the third similarity, a second similarity of the second ultrasound image is obtained.

[0172] As a specific embodiment of this application, the server 12 is further configured to obtain a fourth similarity based on the second ultrasound image; the fourth similarity is the first similarity corresponding to the second ultrasound image; Furthermore, based on the third similarity and the fourth similarity, a second similarity of the second ultrasound image is obtained.

[0173] As a specific embodiment of this application, the server 12 is further configured to extract biomechanical property parameters of soft tissue corresponding to the anesthetic anatomical structure from the static three-dimensional model, wherein the biomechanical property parameters include at least soft tissue density, elastic modulus and Poisson's ratio; Furthermore, under the unified coordinate system, pixel-level registration is performed on the soft tissue regions in the reference ultrasound image and each intraoperative ultrasound image to calculate the deformation displacement vector field of the soft tissue between the reference ultrasound image and each intraoperative ultrasound image. Furthermore, using the rigid constraints of each spatial anchor point of the skeleton in the unified coordinate system as fixed boundary conditions, and combining the biomechanical property parameters and deformation displacement vector field of the soft tissue, a biomechanical constitutive model that conforms to the mechanical properties of human soft tissue is constructed. Furthermore, based on the temporal information of intraoperative ultrasound images, the biomechanical constitutive model is dynamically parameterized to obtain a dynamic three-dimensional model of the anesthetic anatomy.

[0174] The embodiment of the three-dimensional automatic reconstruction system for anesthesia anatomy based on multimodal medical imaging proposed in this application integrates high-resolution rigid structural information from preoperative CT images with real-time soft tissue dynamic information from intraoperative and preoperative ultrasound images. First, a high-precision static three-dimensional model of the anesthesia anatomy is constructed based on the CT images. Then, a unified coordinate system for the multimodal images is established through skeletal spatial anchor point positioning. A reference ultrasound image that best matches the physiological state of the static model is selected. Finally, based on the soft tissue deformation differences between the reference image and the intraoperative ultrasound image, combined with a biomechanical constitutive model, a dynamic three-dimensional model of the anesthesia anatomy is constructed. This system solves the technical problems of mismatch between the static three-dimensional model and the real-time intraoperative anatomical structure, low registration accuracy of multimodal images, and inability to reflect real-time soft tissue deformation in existing technologies. It achieves dynamic three-dimensional visualization reconstruction of anesthesia-related anatomical structures. The reconstructed model can accurately reflect the real-time changes in the patient's intraoperative anatomical structure, providing precise and real-time three-dimensional navigation support for clinical anesthesia operations, effectively improving the safety and accuracy of anesthesia procedures.

[0175] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0176] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the methods, apparatuses, and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or modules may be electrical, mechanical, or other forms.

[0178] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0180] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0181] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video optical disc), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0182] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles of this application.

Claims

1. A three-dimensional automatic reconstruction system for anesthesia anatomy based on multimodal medical imaging, characterized in that, include: A reader is used to acquire the user's medical image data. The medical imaging data includes computed tomography (CT) images and multiple preoperative ultrasound images. The server is used to obtain a static three-dimensional model containing anesthesia anatomical structures based on the medical imaging data information. Furthermore, based on the static three-dimensional model, spatial anchor points are located for the bones in each preoperative ultrasound image to establish a unified coordinate system for multimodal images. Furthermore, based on the rigid constraints of each spatial anchor point in the unified coordinate system, each preoperative ultrasound image is matched to determine the benchmark ultrasound image that is closest to the physiological state of the static three-dimensional model. Furthermore, based on the differences in soft tissue deformation between the baseline ultrasound image and various intraoperative ultrasound images, a dynamic three-dimensional model of the anesthetic anatomy is constructed.

2. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 1, characterized in that, The server is also used to perform preprocessing operations on the computed tomography images and various preoperative ultrasound images, establish a unified pixel coordinate system for each image, and synchronously acquire the spatial pose information of the ultrasound probe corresponding to each preoperative ultrasound image. Furthermore, the preprocessed computed tomography images are automatically identified and segmented using a deep learning segmentation network to obtain the anatomical structure segmentation results of bones, blood vessels, nerves and fascia. Furthermore, based on the anatomical structure segmentation results, a static three-dimensional model containing the anesthetic anatomical structure is generated.

3. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 1, characterized in that, The server is also used to obtain a first similarity score corresponding to each preoperative ultrasound image based on the static three-dimensional model and each preoperative ultrasound image; the first similarity score is at least used to characterize the degree of similarity between the bone region in the preoperative ultrasound image and the corresponding bone projection region in the static three-dimensional model. Furthermore, based on each first similarity, the random sampling consensus algorithm is used to complete the skeletal region traversal matching and obtain the skeletal spatial anchor points between multimodal images; Furthermore, a unified coordinate system for the multimodal image is obtained based on the geometric center of the skeletal spatial anchor point.

4. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 3, characterized in that, The server is also used to obtain a first ultrasound image based on each preoperative ultrasound image; the first ultrasound image is any ultrasound image among the preoperative ultrasound images for which a corresponding first similarity has not been obtained. Furthermore, based on the spatial pose information of the ultrasound probe corresponding to the first ultrasound image, a bone projection image is obtained from the static three-dimensional model; the projection direction of the bone projection image is perpendicular to the imaging plane of the first ultrasound image. Furthermore, based on the first ultrasound image and the bone projection image, a first similarity corresponding to the first ultrasound image is obtained.

5. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 4, characterized in that, The server is further configured to obtain a first edge contour curve and a second edge contour curve based on the first ultrasound image and the bone projection image; the first edge contour curve is the bone edge contour curve in the first ultrasound image; and the second edge contour curve is the bone edge contour curve in the bone projection image. Furthermore, the Hausdorf distance is obtained based on the first edge contour curve and the second edge contour curve; Furthermore, the first similarity is obtained based on the Hausdorff distance; the first similarity is negatively correlated with the Hausdorff distance.

6. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 5, characterized in that, The server is further configured to obtain a first Hu moment vector and a second Hu moment vector based on the first edge contour curve and the second edge contour curve; the first Hu moment vector is the Hu moment vector corresponding to the first edge contour curve; the second Hu moment vector is the Hu moment vector corresponding to the second edge contour curve. Furthermore, based on the first Hu moment vector and the second Hu moment vector, a vector difference is obtained; the vector difference is equal to the absolute value of the difference between the first Hu moment vector and the second Hu moment vector. Furthermore, the first similarity is obtained based on the vector difference and the Hausdorff distance; the first similarity is negatively correlated with the vector difference.

7. The three-dimensional automatic reconstruction system for anesthesia anatomy based on multimodal medical imaging according to any one of claims 1 to 6, characterized in that, The server is also used to obtain a second similarity that corresponds one-to-one with each preoperative ultrasound image based on the rigid constraints of each spatial anchor point in the unified coordinate system; the second similarity is at least used to characterize the degree of similarity between the soft tissue region of the anesthetic anatomical structure in the preoperative ultrasound image and the corresponding soft tissue region in the static three-dimensional model projected in the unified coordinate system. Additionally, the preoperative ultrasound images corresponding to the maximum value of each second similarity are selected and determined as the baseline ultrasound images.

8. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 7, characterized in that, The server is also used to acquire a second ultrasound image based on each preoperative ultrasound image; the second ultrasound image is any ultrasound image among the preoperative ultrasound images for which a corresponding second similarity has not been acquired. Furthermore, based on the spatial pose information of the ultrasound probe corresponding to the second ultrasound image and the unified coordinate system, the soft tissue region of the anesthetic anatomical structure in the static three-dimensional model is projected onto the imaging plane of the second ultrasound image to obtain a soft tissue projection image. Furthermore, based on the soft tissue projection image and the second ultrasound image, a third edge contour curve and a fourth edge contour curve are obtained; the third edge contour curve is the soft tissue edge contour curve in the soft tissue projection image; and the fourth edge contour curve is the soft tissue edge contour curve in the second ultrasound image. Furthermore, a third similarity is obtained based on the third edge contour curve and the fourth edge contour curve; And, based on the third similarity, a second similarity of the second ultrasound image is obtained.

9. The automatic three-dimensional reconstruction system for anesthesia anatomy based on multimodal medical imaging according to claim 8, characterized in that, The server is further configured to obtain a fourth similarity based on the second ultrasound image; the fourth similarity is the first similarity corresponding to the second ultrasound image. Furthermore, based on the third similarity and the fourth similarity, a second similarity of the second ultrasound image is obtained.

10. The three-dimensional automatic reconstruction system for anesthesia anatomy based on multimodal medical imaging according to any one of claims 1 to 6, characterized in that, The server is also used to extract biomechanical property parameters of soft tissue corresponding to the anesthetic anatomical structure from the static three-dimensional model. The biomechanical property parameters include at least soft tissue density, elastic modulus and Poisson's ratio. Furthermore, under the unified coordinate system, pixel-level registration is performed on the soft tissue regions in the reference ultrasound image and each intraoperative ultrasound image to calculate the deformation displacement vector field of the soft tissue between the reference ultrasound image and each intraoperative ultrasound image. Furthermore, using the rigid constraints of each spatial anchor point of the skeleton in the unified coordinate system as fixed boundary conditions, and combining the biomechanical property parameters and deformation displacement vector field of the soft tissue, a biomechanical constitutive model that conforms to the mechanical properties of human soft tissue is constructed. Furthermore, based on the temporal information of intraoperative ultrasound images, the biomechanical constitutive model is dynamically parameterized to obtain a dynamic three-dimensional model of the anesthetic anatomy.

Images