Three-dimensional reconstruction method, device, system, and storage medium

Abstract

The application discloses a three-dimensional reconstruction method, device, system and storage medium. The method comprises the following steps: controlling a dynamic probe and a static imaging array to synchronously collect images to obtain an image set; performing anatomical structure segmentation on a first image and multi-view second images respectively to obtain a first semantic anchor point set corresponding to the first image and a plurality of second semantic anchor point sets corresponding to the multi-view second images; constructing a global semantic map according to the plurality of second semantic anchor point sets and the multi-view second images; determining a current pose of the dynamic probe in a global coordinate system according to the first semantic anchor point set and the global semantic map; performing reconstruction by using a three-dimensional Gaussian splashing model according to the image set, a calibration parameter set and the current pose of the dynamic probe to obtain a current volume video frame of the target region; and outputting the current volume video frame.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and more specifically, to a three-dimensional reconstruction method, apparatus, system, and storage medium. Background Technology

[0002] Minimally invasive surgery, where the procedure is primarily guided by two-dimensional video images provided by endoscopes and other equipment, has become the mainstream of modern surgery. However, two-dimensional images have significant limitations, namely the lack of depth information. This greatly restricts surgeons' ability to perceive anatomical structures in three dimensions, their accuracy in spatial positioning, and their level of fine manipulation. To address this issue, intraoperative three-dimensional reconstruction technology has emerged. Its core purpose is to provide surgeons with a real-time three-dimensional view of the target area, thereby effectively improving the precision and safety of the surgery.

[0003] However, existing intraoperative 3D reconstruction schemes all have significant shortcomings: one type of scheme combines preoperative data with intraoperative images for reconstruction, but its registration accuracy is easily affected by soft tissue deformation and cannot reflect the dynamic changes during the operation in real time; another type of scheme is based entirely on data acquired in real time during the operation for reconstruction, but its field of view and angle of view are severely limited and it is also easily obscured by surgical instruments, blood, etc., resulting in poor reconstruction results. Summary of the Invention

[0004] One objective of this invention is to provide a new technical solution for three-dimensional reconstruction, thereby addressing the technical problems of existing three-dimensional reconstruction technologies, such as low real-time performance, incomplete and unstable reconstruction models, and insufficient accuracy, which fail to meet the needs of complex dynamic surgical scenarios.

[0005] According to a first aspect of the present invention, a three-dimensional reconstruction method is provided, comprising: The dynamic probe and the static imaging array are controlled to synchronously acquire images to obtain an image set; wherein, the image set includes a first image of the target region inside the first user body acquired by the dynamic probe and a multi-view second image of the outside of the first user body acquired by the static imaging array. Anatomical structure segmentation is performed on the first image and the multi-view second image to obtain a first semantic anchor point set corresponding to the first image and multiple second semantic anchor point sets corresponding to the multi-view second image. A global semantic map is constructed based on the multiple sets of second semantic anchor points and the multi-view second images; wherein, the global semantic map includes the three-dimensional coordinates of each anatomical structure of the target region in a global coordinate system; Based on the first set of semantic anchor points and the global semantic map, the current pose of the dynamic probe in the global coordinate system is determined; Based on the image set, calibration parameter set, and the current pose of the dynamic probe, a three-dimensional Gaussian splash model is used for reconstruction to obtain the current volume video frame of the target area; Output the current volume video frame.

[0006] Optionally, the control of the dynamic probe and the static imaging array to synchronously acquire images, resulting in an image set including: Control the start-up of the dynamic probe and the static imaging array, and configure the same acquisition trigger parameters for the dynamic probe and the static imaging array; The acquisition synchronization signal is sent to the dynamic probe and the static imaging array respectively to drive the dynamic probe and the static imaging array to acquire images synchronously, thereby obtaining the image set with acquisition timestamps.

[0007] Optionally, the step of performing anatomical structure segmentation on the first image and the multi-view second image to obtain a first semantic anchor point set corresponding to the first image and multiple second semantic anchor point sets corresponding to the multi-view second image includes: The first image and the multi-view second image are respectively input into the anatomical structure segmentation model to obtain a mask image corresponding to each image; wherein, the mask image includes the confidence score and instance segmentation mask of each anatomical structure in the image; For any image corresponding to a mask, the centroid coordinates of the instance segmentation mask of the target anatomical structure in the mask are extracted and used as the two-dimensional semantic anchor point of the target anatomical structure; wherein, the target anatomical structure is an anatomical structure with a confidence level greater than a confidence threshold. All two-dimensional semantic anchor points corresponding to the first image are aggregated into the first semantic anchor point set, and all two-dimensional semantic anchor points corresponding to any second image are aggregated into the second semantic anchor point set, thus obtaining multiple second semantic anchor point sets corresponding to the multi-view second image.

[0008] Optionally, constructing a global semantic map based on the plurality of second semantic anchor point sets and the multi-view second images includes: The two-dimensional semantic anchor points belonging to the same anatomical structure in the multiple sets of second semantic anchor points are triangulated from multiple perspectives using the extrinsic parameters of the static imaging array to obtain the three-dimensional semantic center point of the anatomical structure in the global coordinate system. A global semantic map is constructed based on the three-dimensional semantic center points of multiple anatomical structures in the global coordinate system.

[0009] Optionally, determining the current pose of the dynamic probe in the global coordinate system based on the first set of semantic anchor points and the global semantic map includes: For each two-dimensional semantic anchor point in the first set of semantic anchor points, a three-dimensional semantic center point corresponding to the two-dimensional semantic anchor point is determined in the global semantic map to obtain the first set of semantic point pairs. Based on the first set of semantic point pairs and the intrinsic parameters of the dynamic probe, the predicted pose of the dynamic probe in the global coordinate system is determined. Based on the predicted pose, the first set of semantic anchor points is back-projected onto the three-dimensional space to obtain the search space range defined on the global semantic map; Within the search space of the global semantic map, find the three-dimensional semantic center point corresponding to the target semantic anchor point in the first semantic anchor point set to obtain the second semantic point pair set; Based on the second set of semantic point pairs, the current pose of the dynamic probe in the global coordinate system is determined.

[0010] Optionally, the step of reconstructing the current volumetric video frame of the target region using a three-dimensional Gaussian splash model based on the image set, the calibration parameter set, and the current pose of the dynamic probe includes: Initialize a 3D Gaussian point set, where each Gaussian point contains position, covariance, color, and opacity parameters; The three-dimensional Gaussian point set is updated through the following iterative optimization steps until the convergence condition is met, and the three-dimensional Gaussian point set that meets the convergence condition is used as the three-dimensional representation of the current volume video frame. The iterative optimization steps include: Based on the intrinsic parameters of the dynamic probe and the current pose of the dynamic probe, the current three-dimensional Gaussian point set is rendered as the first rendered image of the current view of the dynamic probe. For any static imaging unit in the static imaging array, based on the intrinsic and extrinsic parameters of the static imaging unit, the current three-dimensional Gaussian point set is rendered as a second rendered image from the perspective of the static imaging unit. Calculate the first loss value between the first rendered image and the first image, and calculate the first gradient corresponding to each Gaussian point based on the first loss value; For any static imaging unit, calculate the second loss value between the second rendered image and the second image corresponding to the static imaging unit, and calculate the second gradient corresponding to each Gaussian point based on the second loss value; For any Gaussian point in the current three-dimensional Gaussian point set, calculate the number of pixels covered by the Gaussian point on the first rendered image, which is used as the first projected pixel area of ​​the Gaussian point under the current view of the dynamic probe, and calculate the number of pixels covered by the Gaussian point on the second rendered image under the view of any static imaging unit, which is used as the second projected pixel area of ​​the Gaussian point under the view of any static imaging unit. For any Gaussian point, the first average gradient value of the Gaussian point is obtained based on the multiple first gradients accumulated by the Gaussian point within a preset number of iterations, and the second average gradient value of the Gaussian point in the static imaging unit is obtained based on the multiple second gradients accumulated by the Gaussian point within the preset number of iterations corresponding to any static imaging unit. The target gradient value of the Gaussian point is determined based on the first average gradient value, the second average gradient value, the first projected pixel area, and the second projected pixel area. When the target gradient value of a Gaussian point exceeds the gradient threshold, a densification operation is performed on the Gaussian point. Update the parameters of each Gaussian point in the three-dimensional Gaussian point set.

[0011] Optionally, outputting the current volume video frame includes: Receive viewpoint parameters input by the second user; Based on the viewpoint parameters, the three-dimensional Gaussian point set of the current volume video frame is rendered into a target two-dimensional image; The three-dimensional Gaussian point set is reconstructed to generate a target three-dimensional mesh model; Output the two-dimensional image of the target and the three-dimensional mesh model of the target.

[0012] According to a first aspect of the present invention, a three-dimensional reconstruction apparatus is provided, comprising a memory and a processor, the memory being configured to store executable instructions; the processor being configured to operate under the control of the instructions to perform the method as described in the first aspect.

[0013] According to a third aspect of the present invention, a three-dimensional reconstruction system is also provided, comprising a dynamic probe, a static imaging array, and a three-dimensional reconstruction device as described in the second aspect, wherein the three-dimensional reconstruction device is connected to the dynamic probe and the static imaging array respectively, the dynamic probe is used to acquire a first image of a target region inside a user's body and send it to the three-dimensional reconstruction device, and the static imaging array is used to acquire external images covering the target region from multiple angles to obtain a second image with multiple perspectives and send it to the three-dimensional reconstruction device.

[0014] According to a fourth aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method as described in the first aspect.

[0015] One beneficial effect of this invention is that by controlling the dynamic probe and the static imaging array to synchronously acquire images, a first image of the target region inside the first user's body and a second image from multiple perspectives outside the body are simultaneously obtained. This achieves coordinated acquisition of in vivo and in vitro images, avoiding information loss caused by acquiring a single image and laying the foundation for the integrity of the reconstruction model. By performing anatomical structure segmentation on the two types of images respectively, corresponding semantic anchor point sets are obtained. Combined with the second image from multiple perspectives, a global semantic map containing the three-dimensional coordinates of each anatomical structure in the target region is constructed. Then, based on the first semantic anchor point set and the global semantic map, the current pose of the dynamic probe is determined, ensuring accurate probe pose positioning and solving the problems of unstable and insufficient accuracy in the reconstruction model. Finally, by combining the image set, calibration parameter set, and probe current pose, a 3D Gaussian splash model is used for reconstruction and the current volume video frame is output. The 3D Gaussian splash model has the advantages of efficient rendering and accurate representation, which greatly improves the real-time performance of 3D reconstruction. At the same time, through the coordination of in vivo and in vitro images, semantic anchor point positioning, and global map calibration, it makes up for the defects of incomplete, unstable, and insufficient accuracy of existing reconstruction models, and adapts to the core requirements of real-time, complete, and high-precision reconstruction in complex dynamic surgical scenarios, thus effectively solving the technical problems of existing technologies. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0017] Figure 1 This is a schematic diagram of the hardware structure of a three-dimensional reconstruction system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a three-dimensional reconstruction method according to an embodiment of the present invention. Detailed Implementation

[0018] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0019] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0020] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0021] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0023] <Hardware Configuration> Figure 1 This is a schematic diagram of the structure of a three-dimensional reconstruction system 10 according to an embodiment of the present invention.

[0024] like Figure 1 As shown, the three-dimensional reconstruction system 10 includes a dynamic probe 110, a static imaging array 120, and a three-dimensional reconstruction device 100.

[0025] The dynamic probe 110 can be an endoscope, laparoscope, arthroscope, bronchoscope, etc.

[0026] The dynamic probe 110 includes a sterilizable connection interface and at least one dynamic imaging unit. The connection interface is used to connect and fix the dynamic probe 110 to the 3D reconstruction device 100 and transmit a first image to the 3D reconstruction device 100. The sterilizable connection interface can be a connector capable of high-temperature and high-pressure sterilization, and the dynamic imaging unit can be a high-definition imaging module. The first image is an image of a target area within the user's body. The dynamic probe 110 also has an input interface for receiving acquisition synchronization signals from the 3D reconstruction device 100.

[0027] The static imaging array 120 may include at least three static imaging units arranged in a non-collinear manner in a three-dimensional space around the target area to acquire in vitro images covering the entire target area from multiple angles, thereby obtaining a multi-view second image.

[0028] In this embodiment, the static imaging unit can be an industrial camera with 2K resolution and 60fps.

[0029] The static imaging array 120 can be arranged in a surrounding or semi-surrounding manner at a preset position outside the target area (i.e., the surgical area), such as on the operating room shadowless lamp and its matching boom, to form a wide baseline stereoscopic vision layout to completely acquire in vitro images of the target area.

[0030] The 3D reconstruction device 100, as the core processing unit of the system, generates and distributes acquisition synchronization signals to drive the dynamic probe 110 and the static imaging array 120 to acquire images synchronously. It receives the first image transmitted by the dynamic probe 110 (i.e., the image of the target region inside the user's body) and the multi-view second image acquired by the static imaging array 120 (i.e., multi-view images outside the user's body). Combining these with a pre-calibrated set of calibration parameters and the real-time pose of the dynamic probe 110, it performs spatiotemporal registration and 3D fusion reconstruction, ultimately generating complete, accurate, and motion artifact-free volumetric video frames of the target region, providing real-time 3D navigation information for surgical procedures.

[0031] In one embodiment of this application, reference is made to Figure 1 As shown, the three-dimensional reconstruction device 100 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, a synchronization module 1600, a speaker 1700, a microphone 1800, etc.

[0032] Processor 1100 may be a mobile processor. Memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. Interface device 1300 includes, for example, a USB interface, a headphone jack, etc. The interface device is used to connect to the imaging unit of the dynamic probe and static imaging array to receive image data streams. Communication device 1400 is capable of wired or wireless communication. Communication device 1400 may include short-range communication devices, such as any device that performs short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. Communication device 1400 may also include long-range communication devices, such as any device that performs WLAN, GPRS, 2G / 3G / 4G / 5G long-range communication. Display device 1500 is, for example, an LCD screen, a touch screen, etc. Synchronization module 1600 can generate acquisition synchronization signals. Users can input / output voice information through speaker 1700 and microphone 1800.

[0033] In this embodiment, the memory 1200 of the 3D reconstruction apparatus 100 is used to store instructions for controlling the processor 1100 to operate in order to at least execute the 3D reconstruction method according to any embodiment of the present invention. Those skilled in the art can design the instructions according to the disclosed scheme of the present invention. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0034] Despite Figure 1The invention illustrates multiple devices of the three-dimensional reconstruction apparatus 100; however, the invention may refer to only some of these devices. For example, the three-dimensional reconstruction apparatus 100 may refer only to the memory 1200, the processor 1100, and the display device 1500.

[0035] In another embodiment of this application, the three-dimensional reconstruction apparatus 100 includes a memory 1200 and a processor 1100. The memory 1200 is used to store executable instructions; the processor 1100 is used to operate according to the control of the instructions to execute the three-dimensional reconstruction method of the embodiments of this application.

[0036] Taking a 3D reconstruction system in oral surgery (covering implantation, orthognathic surgery, trauma, etc.) as an example, the actual applications of the above components are as follows: The dynamic probe 110 is a customized miniature intraoral dynamic camera adapted to narrow surgical fields. It can be integrated into surgical instruments and detachably connected to a surgical microscope or intraoral mirror via a connector. It is used to capture local tooth and tissue detail images in the surgical area (i.e., the first image). The static imaging array 120 consists of multiple industrial cameras, distributed non-collinearly in a semi-circular shape around the operating table. It serves as a global spatiotemporal coordinate system reference for the extraoral static camera array, simultaneously acquiring global background images of the oral cavity, i.e., multi-view second images. The 3D reconstruction device 100 is a joint reconstruction server that realizes synchronous image acquisition, pose tracking, and 3D reconstruction, ultimately presenting the reconstruction results through a display device.

[0037] <Method Implementation> Figure 2 This is a flowchart illustrating a three-dimensional reconstruction method according to an embodiment of the present invention, which can be implemented by a three-dimensional reconstruction device 100.

[0038] according to Figure 2 As shown, the three-dimensional reconstruction method of this embodiment may include the following steps S2100~S2600: Step S2100: Control the dynamic probe and the static imaging array to synchronously acquire images to obtain an image set; wherein, the image set includes a first image of the target region inside the first user body acquired by the dynamic probe and a multi-view second image of the outside of the first user body acquired by the static imaging array.

[0039] In this embodiment, the first user is the patient who needs to undergo surgery. The target area can be a specific area of ​​the first user that needs to be reconstructed in three dimensions during surgery for the doctor to observe and operate on.

[0040] For example, in oral surgery scenarios (including implantation, orthognathic surgery, trauma, etc.), the target area can be the surgical area consisting of all teeth, jawbone and surrounding soft tissue on the operating table.

[0041] There are several ways to control the synchronous acquisition of images by the dynamic probe and the static imaging array, such as software triggering and synchronization based on network time protocols, which are not limited here.

[0042] The second image from multiple perspectives needs to cover the entire target area.

[0043] In one embodiment of this application, before step S2100 controls the dynamic probe and static imaging array to synchronously acquire images and obtain an image set, the method further includes: obtaining a calibration parameter set.

[0044] In this embodiment, the calibration parameter set includes the intrinsic parameters of the dynamic probe, the intrinsic and extrinsic parameters of the static imaging array (rotation matrix and translation vector of each camera relative to the global coordinate system), and the hand-eye transformation matrix between the dynamic probe and the surgical instruments.

[0045] The set of calibration parameters is pre-stored in the 3D reconstruction device.

[0046] For example, the calibration method for the calibration parameter set is as follows: For the dynamic probe, a fisheye camera model is used. This model is placed in a dedicated cylindrical sterile calibration chamber, and at least 15 sets of calibration images with different insertion depths and rotation angles are acquired. The intrinsic parameters of the dynamic probe are then obtained using a fisheye camera calibration tool. For the static imaging array, at least 20 images of the checkerboard calibration plate in different poses are acquired, and the intrinsic parameters of each camera are solved. Then, using a three-dimensional target, combined with perspective-n-point solving and global bundle adjustment optimization algorithms, the extrinsic parameters of each camera are solved. The calibration of the hand-eye transformation matrix is ​​performed by placing ArUco markers in the common observation area of ​​the dynamic probe and the static imaging array, moving the surgical instruments, and acquiring images in at least 15 different poses. The Zei-Lentz hand-eye calibration algorithm is used to solve the matrix, ultimately yielding the hand-eye transformation matrix.

[0047] Because software-triggered, network time protocol-based synchronization methods result in low synchronization accuracy (often at the millisecond level) and are prone to "exposure-motion" time difference when controlling the synchronous acquisition of images by dynamic probes and static imaging arrays, this application proposes a hardware-level synchronization triggering mechanism to improve the synchronous acquisition accuracy of dynamic probes and static imaging arrays.

[0048] Based on the above, in one embodiment of this application, step S2100 controls the dynamic probe and the static imaging array to synchronously acquire images to obtain an image set, including steps S2100.1 to S2100.2.

[0049] Step S2100.1: Control the start of the dynamic probe and the static imaging array, and configure the same acquisition trigger parameters for the dynamic probe and the static imaging array.

[0050] In this embodiment, firstly, the processor in the 3D reconstruction device sends initialization commands to the dynamic probe and the static imaging array to control the dynamic probe and the static imaging array to complete power-on self-test, imaging module wake-up, and parameter initialization (the dynamic probe loads the fisheye camera calibration parameters, and the static imaging array loads the intrinsic / extrinsic parameter calibration data), ensuring that the imaging functions of both are normal and in a ready-to-acquire state.

[0051] The synchronization module generates acquisition synchronization signals based on Genlock technology. These acquisition synchronization signals can refer to the command signals that drive the imaging unit to initiate image exposure and acquisition. They are hardware-level Genlock synchronization signals, distinct from software trigger signals. Their core advantage is high time accuracy, enabling synchronized operation of multiple devices.

[0052] The acquisition synchronization signal can be a three-level synchronization signal, a black field synchronization signal, etc., and there is no limitation here.

[0053] The processor sends the acquisition trigger parameters (such as trigger frequency, pulse width, and level standard) corresponding to the aforementioned acquisition synchronization signal to the dynamic probe and the static imaging array. The dynamic probe and the static imaging array synchronously write these acquisition trigger parameters into their respective control registers, ensuring that their recognition rules and response thresholds for the trigger signal are completely consistent. At the same time, the dynamic probe and the static imaging unit switch from free-running acquisition mode to externally triggered acquisition mode.

[0054] Step S2100.2: Send the acquisition synchronization signal to the dynamic probe and the static imaging array respectively to drive the dynamic probe and the static imaging array to acquire images synchronously, thereby obtaining the image set with acquisition timestamps.

[0055] In this embodiment, the synchronization module transmits the same acquisition synchronization signal simultaneously to the synchronization phase-locked input interface of the dynamic probe and the synchronization phase-locked input interface of each imaging unit in the static imaging array via a dedicated signal transmission line (such as a BNC line).

[0056] At the same moment they receive the acquisition synchronization signal, the dynamic probe and the static imaging array simultaneously initiate image exposure: the high-definition imaging module of the dynamic probe exposes the target area inside the user's body, acquiring the first image; the static imaging array simultaneously exposes the external surgical area of ​​the target area, acquiring multi-view second images. After each frame of image acquisition is completed, the synchronization module embeds the same high-precision acquisition timestamp into the first image and all second images. This acquisition timestamp is generated based on a hardware clock, with an accuracy down to the microsecond level. Subsequently, the first image with the acquisition timestamp and the multi-view second images are combined to form a complete image set, which is then transmitted to the 3D reconstruction device.

[0057] Step S2200: Perform anatomical structure segmentation on the first image and the multi-view second image respectively to obtain a first semantic anchor point set corresponding to the first image and a plurality of second semantic anchor point sets corresponding to the multi-view second image.

[0058] In this embodiment, the first semantic anchor set contains the identity information and corresponding two-dimensional coordinates of each anatomical structure appearing in the first image, and the second semantic anchor set contains the identity information and corresponding two-dimensional coordinates of each anatomical structure appearing in the multi-view second image.

[0059] For example, in an oral surgery scenario, the first image is specifically a local image of the surgical area acquired by an intraoral dynamic camera, and the multi-view second image is specifically a global image of the entire mouth acquired by an extraoral static camera array.

[0060] Pixel-level contour and gingival margin segmentation of all 32 teeth in the first image and multi-view second images were performed to accurately distinguish the anatomical boundaries of each tooth and assign it unique anatomical identity information. Specifically, the first semantic anchor set is a set of feature points with unique tooth position semantics extracted after segmentation of the first image (intraoral image). The first semantic anchor set can be, for example, feature points corresponding to high-discrimination anatomical sites such as the pits and fissures and cusps of the focused 13th right maxillary third molar and 31st left mandibular lateral incisor in the intraoral image, with each feature point associated with clear tooth position identity information. The second semantic anchor set can be specifically a set of feature points corresponding to the teeth in each multi-view second image (extraoral image) after segmentation. The second semantic anchor set can be, for example, the contour feature points of the 21st right maxillary central incisor and 46th right mandibular first molar in a certain extraoral image. Multiple second semantic anchor sets correspond to the feature point sets with tooth position semantics obtained from the segmentation of multiple second images, and together they provide a semantic association benchmark for subsequent intraoral and extraoral cross-view registration and joint reconstruction.

[0061] In one embodiment of this application, step S2200 performs anatomical structure segmentation on the first image and the multi-view second image respectively to obtain a first semantic anchor point set corresponding to the first image and a plurality of second semantic anchor point sets corresponding to the multi-view second image, including: steps S2200.1 to S2200.3.

[0062] Step S2200.1: Input the first image and the multi-view second image into the anatomical structure segmentation model respectively to obtain the mask image corresponding to each image; The mask image includes the confidence level of each anatomical structure in the image and an instance segmentation mask.

[0063] In this embodiment, different surgical scenarios correspond to different anatomical structure segmentation models.

[0064] For example, in oral surgery scenarios, the anatomical segmentation model is specifically a tooth position detection and segmentation model guided by tooth position query encoding. Based on the RT-DETR architecture, its core innovation lies in replacing the 300 anonymous query vectors commonly used in traditional detectors with 32 expert query vectors that correspond one-to-one with the human permanent dentition. Each expert query vector is assigned fixed anatomical identity information (e.g., the right upper third molar numbered 13) and embeds the corresponding tooth position's anatomical prior information (including the geometric distribution of teeth in the dental arch, adjacency relationships, and morphological features). This transforms the "open set prediction" in general detection into a "closed one-to-one deterministic mapping," completely eliminating ambiguity in label assignment. During the training phase, the model abandons the time-consuming Hungarian matching process and directly establishes constraints between the expert query and the specified target teeth. The loss function integrates weighted binary cross-entropy classification loss (to balance the imbalance of positive and negative samples caused by tooth loss), L1 loss and GIoU (Generalized Intersection over Union Loss) loss for bounding box regression, and Dice (Dice Loss) loss and cross-entropy loss for instance segmentation, achieving accurate end-to-end perception. The model has been trained and optimized with a large number of oral clinical images. The training data covers a variety of complex scenarios such as normal dentition, malocclusion, post-extraction, and post-implantation, enabling it to effectively adapt to image quality degradation caused by motion blur, local occlusion, blood interference, etc. in intraoral dynamic images (first image), as well as perspective distortion caused by facial contour occlusion and tooth projection deformation in extraoral multi-view images (multi-view second image). In actual inference, the input image is processed by the model and outputs a high-quality mask image. Each anatomical structure corresponds to a detection box, a confidence score, and a pixel-level precision instance segmentation mask. The confidence score directly reflects the reliability of the model's recognition results, while the instance segmentation mask accurately delineates the boundaries of each anatomical structure, clearly distinguishing different teeth and surrounding tissues such as teeth, gums, and alveolar bone. This provides a precise anatomical structure localization basis for subsequent two-dimensional semantic anchor point extraction and cross-view registration, fully meeting the stringent requirements of orthognathic surgery, implant surgery, tooth extraction surgery, and other oral surgical procedures for the accuracy and efficiency of anatomical structure recognition.

[0065] Step S2200.2: For any mask image corresponding to an image, extract the centroid coordinates of the instance segmentation mask of the target anatomical structure in the mask image, and use them as the two-dimensional semantic anchor points of the target anatomical structure; The target anatomical structure is an anatomical structure with a confidence level greater than a confidence threshold.

[0066] In this embodiment, for any mask image corresponding to an image, the target anatomical structure with a confidence level greater than a confidence threshold is first determined. The confidence threshold can be a critical value for judging the validity of the anatomical structure detection results, avoiding the impact of false or weak detection results on subsequent processing. The confidence threshold can be, for example, 0.7, 0.8, etc., and can be adjusted according to the surgical accuracy requirements; no limitation is made here.

[0067] Then, the average value of all pixel coordinates within the instance segmentation mask of the target anatomical structure is calculated to obtain the centroid coordinates of the instance segmentation mask of the target anatomical structure. The centroid coordinates of the instance segmentation mask of the target anatomical structure are the geometric center coordinates of the area covered by the instance segmentation mask of the target anatomical structure, reflecting the core position of the target anatomical structure in the image.

[0068] Two-dimensional semantic anchors can be two-dimensional coordinate points with identity information containing anatomical structure (such as specific tooth position ID).

[0069] For example, in oral implant surgery, in the mask image of the first image (i.e., the intraoral image), if the confidence score of the upper right first molar is 0.85 (higher than the confidence threshold of 0.8), then the centroid coordinates of the segmentation mask of that tooth instance (such as image pixel coordinates (320, 480)) are extracted as a two-dimensional semantic anchor point. If the confidence score of a tooth due to occlusion is 0.65 (lower than the confidence threshold of 0.8), then its centroid coordinates are not extracted and it is not used as a semantic anchor point.

[0070] Step S2200.3: Collect all two-dimensional semantic anchor points corresponding to the first image into a first semantic anchor point set, and collect all two-dimensional semantic anchor points corresponding to any second image into a second semantic anchor point set, thereby obtaining multiple second semantic anchor point sets corresponding to the multi-view second image.

[0071] In this embodiment, all two-dimensional semantic anchor points corresponding to the first image (a single frame image from an intraoral dynamic imaging probe) (i.e., anchor points corresponding to all anatomical structures in the intraoral image with qualified confidence and extracted centroid coordinates, such as the three two-dimensional semantic anchor points corresponding to the three teeth visible in the intraoral image) are collected into the first semantic anchor point set. All two-dimensional semantic anchor points corresponding to any second image (a single frame image taken by a camera in an extraoral static imaging array) (anchor points corresponding to all anatomical structures in the extraoral image with qualified confidence and extracted centroid coordinates, such as the eight two-dimensional semantic anchor points corresponding to the eight teeth visible in the extraoral image) are collected into the second semantic anchor point set. Finally, multiple second semantic anchor point sets corresponding to the multi-view second image (a single frame image taken by all cameras in the extraoral static array) are obtained (each single frame image from an extraoral camera corresponds to one second semantic anchor point set).

[0072] Step S2300: Construct a global semantic map based on the plurality of second semantic anchor point sets and the multi-view second image; wherein the global semantic map contains the three-dimensional coordinates of each anatomical structure of the target region in the global coordinate system.

[0073] In this embodiment, the global coordinate system can be a coordinate system with the optical center of the static imaging unit (industrial camera) that is closest to the top of the operating table in the static imaging array (such as the one installed on the main arm of the shadowless lamp) as the origin. The direction of the camera's optical axis pointing vertically downwards towards the operating table surface is the negative Z-axis, the direction parallel to the long side of the operating table and horizontally towards the user's right side is the X-axis, and the Y-axis is determined by the right-hand rule and is parallel to the short side of the operating table and along the direction of the user's head.

[0074] A global coordinate system can unify the spatial position of the dynamic probe and the static imaging array, ensuring the accuracy of pose determination and the spatial consistency of multi-view images.

[0075] In one embodiment of this application, step S2300 constructs a global semantic map based on the plurality of second semantic anchor point sets and the multi-view second image, including: steps S2300.1 to S2300.2.

[0076] Step S2300.1: The two-dimensional semantic anchor points belonging to the same anatomical structure in the multiple sets of second semantic anchor points are triangulated from multiple perspectives using the extrinsic parameters of the static imaging array to obtain the three-dimensional semantic center point of the anatomical structure in the global coordinate system.

[0077] In this embodiment, multi-view triangulation can refer to the process of using the extrinsic parameters of a static imaging array to back-project the two-dimensional semantic anchor points of the same anatomical structure (e.g., the upper right first molar number 16) from different viewpoints onto three-dimensional space, and obtaining the three-dimensional coordinates (i.e., the three-dimensional semantic center point) of the anatomical structure (e.g., the upper right first molar number 16) in the global coordinate system by solving the intersection of multiple sets of projection equations (one projection equation corresponds to one viewpoint).

[0078] For example, in an oral surgery scenario, the three-dimensional semantic center points of all stably detectable teeth can be obtained through step S2300.1.

[0079] Step S2300.2: Construct a global semantic map based on the three-dimensional semantic center points of multiple anatomical structures in the global coordinate system.

[0080] In this embodiment, each anatomical structure corresponds to a three-dimensional semantic center point. For example, in an oral surgery scenario, this three-dimensional reconstruction method might involve 32 anatomical structures, i.e., 32 permanent teeth (or all detectable teeth if any are missing), with each tooth corresponding to a three-dimensional semantic center point. A global semantic map is then obtained based on the three-dimensional semantic center points corresponding to each of the 32 anatomical structures.

[0081] Continuing with the example above, all the three-dimensional semantic center points of the teeth that can be stably detected obtained in step S2300.1 are bound to the corresponding tooth position IDs (such as 13, 31, etc.), and the global sparse point cloud obtained by the extraoral static imaging array through the structure of motion restoration (SfM) algorithm is integrated to form a global semantic map.

[0082] Step S2400: Determine the current pose of the dynamic probe in the global coordinate system based on the first set of semantic anchor points and the global semantic map.

[0083] In one embodiment of this application, step S2400 determines the current pose of the dynamic probe in the global coordinate system based on the first set of semantic anchor points and the global semantic map, including steps S2400.1 to S2400.5.

[0084] Step S2400.1: For each two-dimensional semantic anchor point in the first set of semantic anchor points, determine the three-dimensional semantic center point corresponding to the two-dimensional semantic anchor point in the global semantic map to obtain the first set of semantic point pairs.

[0085] In this embodiment, for any two-dimensional semantic anchor in the first semantic anchor set, based on the identity information carried by the two-dimensional semantic anchor, a three-dimensional semantic center point with the same identity information is searched in the global semantic map. The two-dimensional semantic anchor is then paired with its corresponding three-dimensional semantic center point to obtain a semantic point pair. After all the two-dimensional semantic anchors in the first semantic anchor set are paired, a set of semantic point pairs consisting of "two-dimensional semantic anchor - three-dimensional semantic center point" is obtained, which is the first semantic point pair set.

[0086] For example, in an oral surgery scenario, assuming the first semantic anchor point set contains 5 two-dimensional semantic anchor points, corresponding to tooth positions 11, 12, 21, 22, and 31 respectively (each carrying a corresponding tooth position ID), the tooth position IDs of these 5 two-dimensional semantic anchor points are matched one by one with the tooth position IDs in the global semantic map to find the three-dimensional semantic center points corresponding to tooth positions 11, 12, 21, 22, and 31. Each two-dimensional coordinate (e.g., the two-dimensional coordinate (x1, y1) of tooth position 11) is paired with the corresponding three-dimensional coordinate (e.g., the three-dimensional coordinate (X1, Y1, Z1) of tooth position 11), ultimately forming 5 sets of "two-dimensional-three-dimensional" paired first semantic point pairs.

[0087] Step S2400.2: Determine the predicted pose of the dynamic probe in the global coordinate system based on the first semantic point pair set and the intrinsic parameters of the dynamic probe.

[0088] In this embodiment, the pairing data of "two-dimensional semantic anchor point - three-dimensional semantic center point" in the first semantic point pair set is called, and combined with the intrinsic parameters of the dynamic probe, the initial pose value of the dynamic probe is solved by the PnP (perspective n-point) algorithm. This initial value is the predicted pose of the dynamic probe in the global coordinate system.

[0089] The predicted pose can be a preliminary value of the dynamic probe pose, not the final accurate pose, and contains a certain error, requiring further optimization in subsequent steps.

[0090] The PnP algorithm is an algorithm that solves the camera pose (position and orientation) by knowing the three-dimensional points and their corresponding two-dimensional projection points. It is the core algorithm for solving the predicted pose in this step.

[0091] Continuing with the example above, call the first set of semantic point pairs of the 5 sets of "2D-3D" pairings obtained in step S2400.1, input the intrinsic parameters of the dynamic probe (such as focal length f=500 pixels, principal point coordinates (u0,v0)=(320,240), distortion coefficient k1=-0.01), and solve for the initial position coordinates and rotation angle of the dynamic probe in the global coordinate system through the PnP algorithm, that is, the predicted pose. Assume that the position coordinates of the predicted pose are (X0,Y0,Z0) and the rotation angle is (α0,β0,γ0).

[0092] Step S2400.3: Based on the predicted pose, back-project the first set of semantic anchor points to the three-dimensional space to obtain the search space range defined on the global semantic map.

[0093] In this embodiment, the predicted pose of the dynamic probe in the global coordinate system is used as a reference. Then, using the intrinsic parameters of the dynamic probe, each two-dimensional semantic anchor point in the first semantic anchor point set is back-projected into three-dimensional space to obtain the three-dimensional back-projection point corresponding to each two-dimensional anchor point. A reasonable spatial range is set with these three-dimensional back-projection points as the center (such as a spherical space with the back-projection points as the center and a fixed radius). This range is the search space range defined on the global semantic map.

[0094] The search space can narrow down the search range for subsequent 3D semantic center points, reduce the amount of computation, avoid matching errors, and improve the accuracy of subsequent corresponding point searches.

[0095] Continuing the example above, using the predicted pose obtained in step S2400.2 as a reference, and inputting the dynamic probe intrinsic parameters, the five 2D semantic anchor points in the first semantic anchor point set are back-projected into 3D space, resulting in five 3D back-projection points. A search radius of 2mm is set, and a spherical search space is defined with each 3D back-projection point as its center. The set of all spherical spaces constitutes the search space range on the global semantic map. Subsequent searches will only locate the corresponding 3D semantic center point within this range, avoiding blind searches across the entire global semantic map.

[0096] Step S2400.4: Within the search space of the global semantic map, find the three-dimensional semantic center point corresponding to the target semantic anchor point in the first semantic anchor point set to obtain the second semantic point pair set.

[0097] In this embodiment, target semantic anchors (i.e., two-dimensional semantic anchors with clear imaging, stable mask centroid coordinates, and no obvious noise) are first selected from the first set of semantic anchors, and ambiguous or falsely detected anchors are removed.

[0098] Then, within the search space, based on the identity information of the target semantic anchor point and combined with spatial distance matching rules, the 3D semantic center point that best matches each target semantic anchor point is found. The spatial distance matching rule can refer to calculating the spatial distance between the 3D point projected from the target semantic anchor point and all 3D semantic center points within the search space, and selecting the 3D semantic center point with the smallest distance that meets the distance threshold (e.g., distance ≤ 1mm) as the matching object.

[0099] Finally, all successfully matched “target semantic anchor points - three-dimensional semantic center points” are paired up to form the second semantic point pair set.

[0100] Among them, the second semantic point pair set has higher matching accuracy and less noise compared with the first semantic point pair set, providing reliable data for subsequent accurate pose solving.

[0101] The search space can be a finite space defined on the global semantic map. Target semantic anchors refer to qualified two-dimensional semantic anchors selected from the first set of semantic anchors. Selection criteria include image sharpness, coordinate stability, and lack of occlusion.

[0102] The second semantic point pair set can be a set of pairs of target semantic anchor points and precisely matched three-dimensional semantic center points.

[0103] Continuing the example above, from the five two-dimensional semantic anchors in the first semantic anchor set, four target semantic anchors with clear imaging and stable coordinates are selected (one anchor that is blurred due to oral cavity occlusion is removed). Within the search space, based on the tooth position identifiers of these four target semantic anchors, the spatial distance between the back-projected three-dimensional point of each target semantic anchor and the corresponding three-dimensional semantic center point of the tooth position within the search space is calculated. The three-dimensional semantic center point with the smallest distance that is less than or equal to the distance threshold is selected for pairing, ultimately forming four sets of precisely paired "target semantic anchor-three-dimensional semantic center point", which is the second semantic point pair set.

[0104] Step S2400.5: Determine the current pose of the dynamic probe in the global coordinate system based on the second semantic point pair set.

[0105] In this embodiment, the second semantic point pair set is invoked, combined with the intrinsic parameters of the dynamic probe, and the PnP algorithm combined with the reprojection error optimization method is used to accurately solve and optimize the pose of the dynamic probe. The final pose, which meets the surgical requirements (e.g., reprojection error less than or equal to 0.5 pixels), is used as the current pose of the dynamic probe in the global coordinate system. Simultaneously, this current pose is updated in real time to ensure that the dynamic probe continuously provides accurate spatial positioning as it moves during the operation.

[0106] Reprojection error optimization can be achieved by projecting the 3D semantic center point onto the 2D image of the dynamic probe based on the pose of the dynamic probe obtained by the PnP algorithm, calculating the pixel distance (reprojection error) between the projected point and the corresponding target semantic anchor point, and iteratively optimizing and adjusting the pose parameters of the dynamic probe to reduce the reprojection error to below a preset threshold, thereby improving pose accuracy.

[0107] Continuing with the example above, the four sets of second semantic point pairs obtained in step S2400.4 are called, and the dynamic probe intrinsic parameters are input. The PnP algorithm is used to solve for the initial pose of the dynamic probe. Then, the pose parameters are iteratively adjusted using the reprojection error optimization method, optimizing the reprojection error from the initial 1.2 pixels to 0.3 pixels (satisfying the surgical preset threshold of less than or equal to 0.5 pixels). Finally, the current pose of the dynamic probe in the global coordinate system is obtained. When the dynamic probe moves in the oral cavity, steps S2400.1 to S2400.5 are repeated to continuously update the current pose, ensuring the real-time performance and accuracy of the positioning.

[0108] Step S2500: Based on the image set, calibration parameter set, and the current pose of the dynamic probe, a three-dimensional Gaussian splash model is used to reconstruct the current volume video frame of the target area.

[0109] In this embodiment, a volumetric video frame is a frame in a volumetric video stream. The volumetric video frame records the geometric and appearance information of a dynamic 3D scene at a certain moment in the form of 3D voxels, point clouds, or meshes.

[0110] In one embodiment of this application, step S2500 reconstructs the current volume video frame of the target region using a three-dimensional Gaussian splash model based on the image set, the calibration parameter set, and the current pose of the dynamic probe, including steps S2500.1 to S2500.2.

[0111] Step S2500.1: Initialize a three-dimensional Gaussian point set, where each Gaussian point contains position, covariance, color, and opacity parameters.

[0112] Step S2500.2: The three-dimensional Gaussian point set is updated through the following iterative optimization steps S1 to S9 until the convergence condition is met, and the three-dimensional Gaussian point set that meets the convergence condition is used as the three-dimensional representation of the current volume video frame.

[0113] In this embodiment, the goal of iterative optimization is to make the rendering result of the 3D Gaussian point set as consistent as possible with the actual acquired image, and finally obtain an accurate 3D representation of the target region.

[0114] The convergence condition can be a criterion for stopping the iteration. Specifically, it can be that the change in the Gaussian point parameter is less than a preset threshold, or that the loss value tends to stabilize, etc. No specific limitation is made here.

[0115] The iterative optimization steps include steps S1 to S9.

[0116] Step S1: Based on the intrinsic parameters of the dynamic probe and the current pose of the dynamic probe, render the current three-dimensional Gaussian point set as the first rendered image of the current viewpoint of the dynamic probe.

[0117] In this embodiment, the current three-dimensional Gaussian point set is used to render the first rendered image corresponding to the current view of the dynamic probe, based on the intrinsic parameters of the dynamic probe and the current pose of the dynamic probe in the global coordinate system.

[0118] Step S2: For any static imaging unit in the static imaging array, based on the intrinsic and extrinsic parameters of the static imaging unit, render the current three-dimensional Gaussian point set as the second rendered image from the perspective of the static imaging unit.

[0119] In this embodiment, for each static imaging unit in the static imaging array, the current three-dimensional Gaussian point set is rendered to obtain the second rendered image from the perspective of the static imaging unit based on the intrinsic and extrinsic parameters of the static imaging unit.

[0120] Step S3: Calculate the first loss value between the first rendered image and the first image, and calculate the first gradient corresponding to each Gaussian point based on the first loss value.

[0121] In this embodiment, the first loss value can be a quantitative indicator of the difference (color difference and position difference) between the first rendered image and the first image. The first loss value between the first rendered image and the first image is calculated using a loss function (i.e., the larger the loss value, the greater the deviation).

[0122] Then, based on this first loss value, the first gradient of the Gaussian point location parameters is calculated in reverse. The direction and magnitude of the first gradient of the Gaussian point reflect the direction and extent to which adjusting the Gaussian point location reduces the loss. The first gradient can provide a basis for subsequent updates of the Gaussian point parameters.

[0123] Step S4: For any static imaging unit, calculate the second loss value between the second rendered image and the second image corresponding to the static imaging unit, and calculate the second gradient corresponding to each Gaussian point based on the second loss value.

[0124] This step is basically the same as the method used in step S4 above to calculate the first loss value, and will not be elaborated here.

[0125] Step S5: For any Gaussian point in the current three-dimensional Gaussian point set, calculate the number of pixels covered by the Gaussian point on the first rendered image, which is taken as the first projected pixel area of ​​the Gaussian point under the current view of the dynamic probe, and calculate the number of pixels covered by the Gaussian point on the second rendered image under the view of any static imaging unit, which is taken as the second projected pixel area of ​​the Gaussian point under the view of any static imaging unit.

[0126] In this embodiment, the projection percentage (i.e., projected pixel area) of each Gaussian point in the current 3D Gaussian point set under different viewpoints is statistically analyzed to provide a weighting basis for subsequent gradient weighting. The larger the projected pixel area, the greater the influence of the Gaussian point under the corresponding viewpoint, and the higher the weight should be assigned in subsequent calculations.

[0127] Step S6: For any Gaussian point, obtain the first average gradient value of the Gaussian point based on the multiple first gradients accumulated by the Gaussian point within a preset number of iterations, and obtain the second average gradient value of the Gaussian point in the static imaging unit based on the multiple second gradients accumulated by the Gaussian point in the preset number of iterations corresponding to any static imaging unit.

[0128] In this embodiment, the preset number of iterations can be 10 times, etc., and is not limited here.

[0129] Step S7: Determine the target gradient value of the Gaussian point based on the first average gradient value, the second average gradient value, the first projected pixel area, and the second projected pixel area.

[0130] In this embodiment, the first projected pixel area corresponding to the Gaussian point and the second projected pixel area of ​​each viewpoint are used as weights to perform a weighted summation of the first average gradient value corresponding to the Gaussian point and the second average gradient value corresponding to each viewpoint, and the weighted average position gradient is obtained as the target gradient value of the Gaussian point.

[0131] The larger the projected pixel area, the higher the weight, which means that the gradient at that viewpoint has a greater impact on the adjustment of the Gaussian point parameters.

[0132] The target gradient value is a gradient value that integrates gradient information from multiple perspectives and is weighted according to the contribution of the perspective (projected pixel area), which can solve the problem of extreme imbalance in gradient contribution.

[0133] Step S8: When the target gradient value of the Gaussian point exceeds the gradient threshold, a densification operation is performed on the Gaussian point.

[0134] In this embodiment, it is a step to achieve "on-demand densification" of the three-dimensional Gaussian point set. The purpose is to allow the key details of the target area (areas with large gradient deviations) to obtain more Gaussian point coverage and improve the reconstruction accuracy.

[0135] Specifically, the target gradient value is compared with a preset gradient threshold. If the target gradient value exceeds the threshold, it indicates that the reconstruction of the region corresponding to the Gaussian point has a large deviation and lacks detail, requiring a densification operation. This densification operation can involve splitting the Gaussian point, increasing the number of Gaussian points. The densification operation ensures the reconstruction accuracy of the region.

[0136] If the target gradient value does not exceed the gradient threshold, densification will not be performed to avoid unnecessary computational overhead.

[0137] Step S9: Update the parameters of each Gaussian point in the three-dimensional Gaussian point set.

[0138] In this embodiment, based on the gradients calculated in steps S3 and S4 (or the target gradient value in step S7), the position, covariance, color, and opacity of each Gaussian point are adjusted to minimize the deviation between the rendered image corresponding to the adjusted Gaussian point and the real image. Simultaneously, a densification operation is performed in step S8, and the newly obtained Gaussian points after splitting are added to the Gaussian point set to update the overall structure of the Gaussian point set, completing the parameter update for a single iteration and laying the foundation for the next iteration.

[0139] Step S2600: Output the current volume video frame.

[0140] In this embodiment, the current body video frame is output to the surgeon, and / or the current body video frame is output to a third party (i.e., the patient's family member), which is not limited here.

[0141] It should be noted that steps S2100 to S2600 above can be performed on a time series to generate multiple volume video frames and obtain a volume video stream.

[0142] In one embodiment of this application, step S2600, outputting the current volume video frame, includes: Steps S2600.1 to S2600.4.

[0143] Step S2600.1: Receive the viewpoint parameters input by the second user.

[0144] In this embodiment, the second user can be a surgeon, a patient's family member, etc., and there is no limitation here.

[0145] Viewpoint parameters refer to the core parameters used to determine the viewing angle. Viewpoint parameters can include viewing position, viewing angle, zoom level, etc., but are not limited here.

[0146] Step S2600.2: Render the three-dimensional Gaussian point set of the current volume video frame into a target two-dimensional image according to the viewpoint parameters.

[0147] In this embodiment, the three-dimensional Gaussian point set is converted into a two-dimensional image recognizable by human vision according to the viewpoint parameters, thus obtaining the target two-dimensional image. The target two-dimensional image can clearly present the details of the target area from the specified viewpoint (in the case of the oral cavity, it can present the shape of teeth, the position of surgical instruments, etc.), thereby facilitating the observation of the target area by the second user.

[0148] Step S2600.3: Perform surface reconstruction on the three-dimensional Gaussian point set to generate the target three-dimensional mesh model.

[0149] In this embodiment, surface reconstruction can be a process of generating a continuous and complete three-dimensional surface by fitting the geometric information such as the spatial position and covariance contained in the three-dimensional Gaussian point set through an algorithm, which can make up for the lack of structural integrity of the discrete Gaussian point set.

[0150] The target 3D mesh model can refer to a 3D model composed of vertices, edges, and faces obtained after surface reconstruction. It is a concrete representation of the 3D structure of the target region. Compared with a 3D Gaussian point set, it is more intuitive and operable and can be used in surgical planning, postoperative evaluation, and other fields.

[0151] Step S2600.4: Output the target two-dimensional image and the target three-dimensional mesh model.

[0152] Based on the above, by controlling the dynamic probe and static imaging array to synchronously acquire images, a first image of the target region inside the first user's body and a second image from multiple perspectives outside the body are simultaneously obtained. This achieves coordinated acquisition of in vivo and in vitro images, avoiding information loss caused by acquiring a single image and laying the foundation for the integrity of the reconstruction model. By segmenting the two types of images into anatomical structures, corresponding semantic anchor point sets are obtained. Combined with the second image from multiple perspectives, a global semantic map containing the three-dimensional coordinates of each anatomical structure in the target region is constructed. Then, based on the first semantic anchor point set and the global semantic map, the current pose of the dynamic probe is determined, ensuring accurate probe pose localization. The system is accurate, addressing the issues of unstable and insufficient precision in reconstruction models. Finally, by combining image sets, calibration parameter sets, and the current probe pose, a 3D Gaussian splash model is used for reconstruction, outputting the current volume video frame. The 3D Gaussian splash model has the advantages of efficient rendering and accurate representation, significantly improving the real-time performance of 3D reconstruction. At the same time, through the combined efforts of in vivo and in vitro image collaboration, semantic anchor point localization, and global map calibration, it overcomes the shortcomings of incomplete, unstable, and insufficient precision in existing reconstruction models, adapting to the core requirements of real-time, complete, and high-precision reconstruction in complex and dynamic surgical scenarios, thus effectively solving the technical problems of existing technologies.

[0153] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above method embodiments.

[0154] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0155] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0156] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0157] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0158] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0159] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0160] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0162] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: The dynamic probe and the static imaging array are controlled to synchronously acquire images to obtain an image set; wherein, the image set includes a first image of the target region inside the first user body acquired by the dynamic probe and a multi-view second image of the outside of the first user body acquired by the static imaging array. Anatomical structure segmentation is performed on the first image and the multi-view second image to obtain a first semantic anchor point set corresponding to the first image and multiple second semantic anchor point sets corresponding to the multi-view second image. A global semantic map is constructed based on the multiple sets of second semantic anchor points and the multi-view second images; wherein, the global semantic map includes the three-dimensional coordinates of each anatomical structure of the target region in a global coordinate system; Based on the first set of semantic anchor points and the global semantic map, the current pose of the dynamic probe in the global coordinate system is determined; Based on the image set, calibration parameter set, and the current pose of the dynamic probe, a three-dimensional Gaussian splash model is used for reconstruction to obtain the current volume video frame of the target area; Output the current volume video frame.

2. The method according to claim 1, characterized in that, The control of the dynamic probe and the static imaging array to synchronously acquire images results in an image set including: Control the start-up of the dynamic probe and the static imaging array, and configure the same acquisition trigger parameters for the dynamic probe and the static imaging array; The acquisition synchronization signal is sent to the dynamic probe and the static imaging array respectively to drive the dynamic probe and the static imaging array to acquire images synchronously, thereby obtaining the image set with acquisition timestamps.

3. The method according to claim 1, characterized in that, The step of performing anatomical structure segmentation on the first image and the multi-view second image respectively to obtain a first semantic anchor point set corresponding to the first image and multiple second semantic anchor point sets corresponding to the multi-view second image includes: The first image and the multi-view second image are respectively input into the anatomical structure segmentation model to obtain a mask image corresponding to each image; wherein, the mask image includes the confidence score and instance segmentation mask of each anatomical structure in the image; For any image corresponding to a mask, the centroid coordinates of the instance segmentation mask of the target anatomical structure in the mask are extracted and used as the two-dimensional semantic anchor point of the target anatomical structure; wherein, the target anatomical structure is an anatomical structure with a confidence level greater than a confidence threshold. All two-dimensional semantic anchor points corresponding to the first image are aggregated into the first semantic anchor point set, and all two-dimensional semantic anchor points corresponding to any second image are aggregated into the second semantic anchor point set, thus obtaining multiple second semantic anchor point sets corresponding to the multi-view second image.

4. The method according to claim 1, characterized in that, The step of constructing a global semantic map based on the plurality of second semantic anchor point sets and the multi-view second images includes: The two-dimensional semantic anchor points belonging to the same anatomical structure in the multiple sets of second semantic anchor points are triangulated from multiple perspectives using the extrinsic parameters of the static imaging array to obtain the three-dimensional semantic center point of the anatomical structure in the global coordinate system. A global semantic map is constructed based on the three-dimensional semantic center points of multiple anatomical structures in the global coordinate system.

5. The method according to claim 1, characterized in that, The step of determining the current pose of the dynamic probe in the global coordinate system based on the first set of semantic anchor points and the global semantic map includes: For each two-dimensional semantic anchor point in the first set of semantic anchor points, a three-dimensional semantic center point corresponding to the two-dimensional semantic anchor point is determined in the global semantic map to obtain the first set of semantic point pairs. Based on the first set of semantic point pairs and the intrinsic parameters of the dynamic probe, the predicted pose of the dynamic probe in the global coordinate system is determined. Based on the predicted pose, the first set of semantic anchor points is back-projected into three-dimensional space to obtain the search space range defined on the global semantic map; Within the search space of the global semantic map, find the three-dimensional semantic center point corresponding to the target semantic anchor point in the first semantic anchor point set to obtain the second semantic point pair set; Based on the second set of semantic point pairs, the current pose of the dynamic probe in the global coordinate system is determined.

6. The method according to claim 1, characterized in that, The step of reconstructing the current volumetric video frame of the target region using a three-dimensional Gaussian splash model based on the image set, the calibration parameter set, and the current pose of the dynamic probe includes: Initialize a 3D Gaussian point set, where each Gaussian point contains position, covariance, color, and opacity parameters; The three-dimensional Gaussian point set is updated through the following iterative optimization steps until the convergence condition is met, and the three-dimensional Gaussian point set that meets the convergence condition is used as the three-dimensional representation of the current volume video frame. The iterative optimization steps include: Based on the intrinsic parameters of the dynamic probe and the current pose of the dynamic probe, the current three-dimensional Gaussian point set is rendered as the first rendered image of the current view of the dynamic probe. For any static imaging unit in the static imaging array, based on the intrinsic and extrinsic parameters of the static imaging unit, the current three-dimensional Gaussian point set is rendered as a second rendered image from the perspective of the static imaging unit. Calculate the first loss value between the first rendered image and the first image, and calculate the first gradient corresponding to each Gaussian point based on the first loss value; For any static imaging unit, calculate the second loss value between the second rendered image and the second image corresponding to the static imaging unit, and calculate the second gradient corresponding to each Gaussian point based on the second loss value; For any Gaussian point in the current three-dimensional Gaussian point set, calculate the number of pixels covered by the Gaussian point on the first rendered image, which is used as the first projected pixel area of ​​the Gaussian point under the current view of the dynamic probe, and calculate the number of pixels covered by the Gaussian point on the second rendered image under the view of any static imaging unit, which is used as the second projected pixel area of ​​the Gaussian point under the view of any static imaging unit. For any Gaussian point, the first average gradient value of the Gaussian point is obtained based on the multiple first gradients accumulated by the Gaussian point within a preset number of iterations, and the second average gradient value of the Gaussian point in the static imaging unit is obtained based on the multiple second gradients accumulated by the Gaussian point within the preset number of iterations corresponding to any static imaging unit. The target gradient value of the Gaussian point is determined based on the first average gradient value, the second average gradient value, the first projected pixel area, and the second projected pixel area. When the target gradient value of a Gaussian point exceeds the gradient threshold, a densification operation is performed on the Gaussian point. Update the parameters of each Gaussian point in the three-dimensional Gaussian point set.

7. The method according to claim 1, characterized in that, The output of the current volume video frame includes: Receive viewpoint parameters input by the second user; Based on the viewpoint parameters, the three-dimensional Gaussian point set of the current volume video frame is rendered into a target two-dimensional image; The three-dimensional Gaussian point set is reconstructed to generate a target three-dimensional mesh model; Output the two-dimensional image of the target and the three-dimensional mesh model of the target.

8. A three-dimensional reconstruction device, characterized in that, It includes a memory and a processor, the memory being used to store executable instructions; the processor being used to operate under the control of the instructions to perform the method as described in any one of claims 1 to 7.

9. A three-dimensional reconstruction system, characterized in that, The device includes a dynamic probe, a static imaging array, and a three-dimensional reconstruction device as described in claim 8. The three-dimensional reconstruction device is connected to the dynamic probe and the static imaging array, respectively. The dynamic probe is used to acquire a first image of a target region inside the user's body and send it to the three-dimensional reconstruction device. The static imaging array is used to acquire external images covering the target region from multiple angles to obtain a second image with multiple perspectives and send it to the three-dimensional reconstruction device.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images