Endoscopic surgery augmented reality display method, electronic equipment and storage medium

Abstract

The invention provides an augmented reality display method for endoscopic surgery, electronic equipment and a storage medium, and is applied to the technical field of image processing. The method comprises the following steps: acquiring a current frame image of a target part in an operation process; inputting the current frame image into a pre-trained projection attitude estimation model to obtain a camera attitude estimation parameter corresponding to the current frame image, the camera attitude estimation parameter representing a position of a standard statistical shape model corresponding to the target part in a camera coordinate system corresponding to the current frame image, the standard statistical shape model is used for capturing a shape statistical rule of the target part and does not depend on an anatomical mark of the target part; and based on the camera attitude estimation parameters, rendering and superposing the pre-constructed target three-dimensional part model corresponding to the target part into the current frame image, and generating an augmented reality display result of the target part, thereby achieving an effect of effectively performing high-precision augmented reality display on a part with fewer anatomical markers.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an augmented reality display method, electronic device and storage medium for laparoscopic surgery. Background Technology

[0002] Augmented Reality (AR) technology, as an advanced computer vision technology, is combined with image processing and 3D reconstruction technologies to register and fuse the 3D model of the patient's preoperative reconstructed body part with the intraoperative endoscopic image. This achieves precise alignment between preoperative images and real-time intraoperative images, providing doctors with visual information about hidden anatomical structures and significantly improving surgical accuracy and safety.

[0003] In related technologies, intraoperative endoscopic images and preoperative 3D models of the surgical site are typically registered based on feature points or contours of the site, thereby achieving augmented reality display of the site under laparoscopic vision. This method has been successfully applied in surgeries on sites such as the liver and uterus. However, in surgeries on sites with extremely limited anatomical structures, such as the kidneys, the lack of visible anatomical landmarks for registration constraints leads to poor registration accuracy, making it difficult to achieve high-precision augmented reality display and failing to meet the precision and safety requirements of the surgery.

[0004] Therefore, there is an urgent need for a high-precision augmented reality display solution that can effectively perform surgery on areas with few anatomical landmarks. Summary of the Invention

[0005] This application provides an augmented reality display method, electronic device, and storage medium for laparoscopic surgery, which can achieve a high-precision augmented reality display effect for surgery on areas with few anatomical landmarks.

[0006] In a first aspect, this application provides an augmented reality display method for laparoscopic surgery, comprising:

[0007] Acquire the current frame image of the target area during the surgical procedure;

[0008] The current frame image is input into the pre-trained projection pose estimation model to obtain the camera pose estimation parameters corresponding to the current frame image. The camera pose estimation parameters represent the position of the standard statistical shape model corresponding to the target part in the camera coordinate system corresponding to the current frame image. The standard statistical shape model is used to capture the shape statistical regularity of the target part and does not depend on the anatomical landmarks of the target part.

[0009] Based on camera pose estimation parameters, the pre-constructed target 3D part model corresponding to the target part is rendered and superimposed onto the current frame image to generate the augmented reality display result of the target part. The target 3D part model is a specific 3D part model that has been registered to the standard statistical shape model space.

[0010] In one possible implementation, the standard statistical shape model is constructed as follows:

[0011] Acquire multiple sample medical images of the target area, including medical images of the target area when it is normal and medical images of the target area when it is abnormal;

[0012] The three-dimensional contour of each sample medical image is extracted from multiple sample medical images to obtain the first three-dimensional part model corresponding to each sample medical image;

[0013] Based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape.

[0014] Based on the average shape and multiple second-dimensional part models, principal component analysis of shape is performed to obtain the standard statistical shape model corresponding to the target part.

[0015] In one possible implementation, based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape, including:

[0016] Multiple first three-dimensional part models are preprocessed to obtain multiple preprocessed first three-dimensional part models. The preprocessing includes mesh resampling, topology repair and surface smoothing.

[0017] Based on the least squares method, the preprocessed first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and the multiple second three-dimensional part models aligned with the average shape.

[0018] In one possible implementation, the training dataset used during training of the projection pose estimation model is obtained as follows:

[0019] The keyframe sample images are obtained from the sample video stream of the target area during the operation. The keyframe sample images cover different viewpoints and different stages of the operation.

[0020] For each keyframe sample image in multiple keyframe sample images, the keyframe sample image is contour-fitted with the third three-dimensional part model of the target part constructed before surgery to obtain the first spatial transformation matrix of the third three-dimensional part model in the camera coordinate system corresponding to the keyframe sample image. The third three-dimensional part model is a three-dimensional part model registered to the standard statistical shape model space.

[0021] Based on the first spatial transformation matrix and the pre-constructed second spatial transformation matrix, the camera pose parameters of the standard statistical shape model in the camera coordinate system are determined. The second spatial transformation matrix represents the spatial projection position transformation relationship from the fourth three-dimensional part model to the standard statistical shape model. The fourth three-dimensional part model is a three-dimensional part model that has not been registered to the standard statistical shape model space.

[0022] The camera intrinsic parameters of multiple keyframe sample images are normalized to standard intrinsic parameters;

[0023] The keyframe sample images, standard intrinsic parameters, and camera pose parameters are integrated to form a training dataset.

[0024] In one possible implementation, the second spatial transformation matrix is ​​obtained as follows:

[0025] By applying the least squares method, based on the centroid and principal axis positions of the standard statistical shape model, coarse registration is performed between the fourth three-dimensional part model and the standard statistical shape model to obtain the coarse registration spatial transformation matrix and the coarse registration fourth three-dimensional part model.

[0026] By applying the least squares method, based on deformation estimation and iterative optimization, fine registration is performed on the coarsely registered fourth three-dimensional part model and the standard statistical shape model to obtain the fine registration spatial transformation matrix.

[0027] The product of the coarse registration spatial transformation matrix and the fine registration spatial transformation matrix is ​​determined as the second spatial transformation matrix.

[0028] In one possible implementation, the loss function of the projection pose estimation model includes a pose estimation loss function and a shape loss function.

[0029] In one possible implementation, acquiring the current frame image of the target site during the surgical procedure includes:

[0030] Obtain raw images of the target area during the surgical procedure;

[0031] By applying a preset standard intrinsic parameter matrix, the original image is processed to perform camera intrinsic parameter normalization to obtain the current frame image of the target area during the surgical procedure.

[0032] Secondly, this application provides an augmented reality display device for laparoscopic surgery, comprising:

[0033] The acquisition module is used to acquire the current frame image of the target area during the surgical procedure;

[0034] The input module is used to input the current frame image into the pre-trained projection pose estimation model to obtain the camera pose estimation parameters corresponding to the current frame image. The camera pose estimation parameters represent the position of the standard statistical shape model corresponding to the target part in the camera coordinate system corresponding to the current frame image. The standard statistical shape model is used to capture the shape statistical regularity of the target part and does not depend on the anatomical landmarks of the target part.

[0035] The display enhancement module is used to render and overlay the pre-built target 3D part model corresponding to the target part onto the current frame image based on the camera pose estimation parameters, thereby generating an augmented reality display result of the target part. The target 3D part model is a specific 3D part model that has been registered to the standard statistical shape model space.

[0036] In one possible implementation, the standard statistical shape model is constructed as follows:

[0037] Acquire multiple sample medical images of the target area, including medical images of the target area when it is normal and medical images of the target area when it is abnormal;

[0038] The three-dimensional contour of each sample medical image is extracted from multiple sample medical images to obtain the first three-dimensional part model corresponding to each sample medical image;

[0039] Based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape.

[0040] Based on the average shape and multiple second-dimensional part models, principal component analysis of shape is performed to obtain the standard statistical shape model corresponding to the target part.

[0041] In one possible implementation, based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape, including:

[0042] Multiple first three-dimensional part models are preprocessed to obtain multiple preprocessed first three-dimensional part models. The preprocessing includes mesh resampling, topology repair and surface smoothing.

[0043] Based on the least squares method, the preprocessed first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and the multiple second three-dimensional part models aligned with the average shape.

[0044] In one possible implementation, the training dataset used during training of the projection pose estimation model is obtained as follows:

[0045] The keyframe sample images are obtained from the sample video stream of the target area during the operation. The keyframe sample images cover different viewpoints and different stages of the operation.

[0046] For each keyframe sample image in multiple keyframe sample images, the keyframe sample image is contour-fitted with the third three-dimensional part model of the target part constructed before surgery to obtain the first spatial transformation matrix of the third three-dimensional part model in the camera coordinate system corresponding to the keyframe sample image. The third three-dimensional part model is a three-dimensional part model registered to the standard statistical shape model space.

[0047] Based on the first spatial transformation matrix and the pre-constructed second spatial transformation matrix, the camera pose parameters of the standard statistical shape model in the camera coordinate system are determined. The second spatial transformation matrix represents the spatial projection position transformation relationship from the fourth three-dimensional part model to the standard statistical shape model. The fourth three-dimensional part model is a three-dimensional part model that has not been registered to the standard statistical shape model space.

[0048] The camera intrinsic parameters of multiple keyframe sample images are normalized to standard intrinsic parameters;

[0049] The keyframe sample images, standard intrinsic parameters, and camera pose parameters are integrated to form a training dataset.

[0050] In one possible implementation, the second spatial transformation matrix is ​​obtained as follows:

[0051] By applying the least squares method, based on the centroid and principal axis positions of the standard statistical shape model, coarse registration is performed between the fourth three-dimensional part model and the standard statistical shape model to obtain the coarse registration spatial transformation matrix and the coarse registration fourth three-dimensional part model.

[0052] By applying the least squares method, based on deformation estimation and iterative optimization, fine registration is performed on the coarsely registered fourth three-dimensional part model and the standard statistical shape model to obtain the fine registration spatial transformation matrix.

[0053] The product of the coarse registration spatial transformation matrix and the fine registration spatial transformation matrix is ​​determined as the second spatial transformation matrix.

[0054] In one possible implementation, the loss function of the projection pose estimation model includes a pose estimation loss function and a shape loss function.

[0055] In one possible implementation, acquiring the current frame image of the target site during the surgical procedure includes:

[0056] Obtain raw images of the target area during the surgical procedure;

[0057] By applying a preset standard intrinsic parameter matrix, the original image is processed to perform camera intrinsic parameter normalization to obtain the current frame image of the target area during the surgical procedure.

[0058] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0059] The memory stores instructions that the computer executes;

[0060] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0061] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0062] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0063] The augmented reality display method, electronic device, and storage medium for laparoscopic surgery provided in this application acquire the current frame image of the target site during the surgical process; input the current frame image into a pre-trained projection pose estimation model to obtain the camera pose estimation parameters corresponding to the current frame image. The camera pose estimation parameters characterize the position of the standard statistical shape model corresponding to the target site in the camera coordinate system corresponding to the current frame image. The standard statistical shape model is used to capture the shape statistical regularity of the target site and does not depend on the anatomical landmarks of the target site; based on the camera pose estimation parameters, the pre-constructed target three-dimensional part model corresponding to the target site is rendered and superimposed onto the current frame image to generate the augmented reality display result of the target site. The target three-dimensional part model is a specific three-dimensional part model registered to the standard statistical shape model space. This application introduces a standard statistical shape model constructed based on group anatomical knowledge as a universal shape prior, which can capture the statistical regularity and variation pattern of the shape of the part. On this basis, the target three-dimensional part model that has been registered to the standard statistical shape model space is used. That is, the specific three-dimensional part models of different objects are unified into the same standard space through registration, eliminating the specificity of the target parts of different objects. When rendering and superimposing the current frame image, it is not necessary to rely on the anatomical landmarks on the surface of the target part. Instead, the shape of the target three-dimensional part model is constrained by the group anatomical knowledge, realizing the alignment between the preoperative target three-dimensional part model and the current frame image during surgery. This significantly improves the registration accuracy, thereby improving the accuracy of augmented reality display, providing doctors with stable augmented reality navigation, reducing the risk of intraoperative errors, and improving the success rate of surgery. Furthermore, by leveraging the complex nonlinear mapping relationship learned by the end-to-end neural network from endoscopic images to the camera pose in the standard statistical shape model space, the intraoperative endoscopic images are processed, and the camera pose estimation parameters corresponding to the current frame image contained in the endoscopic image are output in real time. These camera pose estimation parameters, constrained by the global shape prior of the standard statistical shape model, conform to anatomical laws and are used to accurately back-map the preoperative target 3D model to the target position in the current frame image, achieving a direct mapping from the 3D model to the endoscopic image. This effectively avoids registration failures caused by dependence on local and variable anatomical landmarks, improving real-time performance and accuracy. In addition, since the projection pose estimation model uses the standard statistical shape model, which is universal and does not depend on specific individual anatomical landmarks, as a unified geometric reference system, this projection pose estimation model is applicable to any surgical object that conforms to this statistical law, solving the problem in related technologies that require customized training models for each specific object, and greatly improving the system's versatility. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] Figure 1 A schematic diagram of a scene for an augmented reality display method for laparoscopic surgery provided in an embodiment of this application;

[0066] Figure 2 A schematic flowchart of the augmented reality display method for laparoscopic surgery provided in the embodiments of this application;

[0067] Figure 3 A flowchart illustrating the construction method of the standard statistical shape model provided in the embodiments of this application;

[0068] Figure 4 A flowchart illustrating an augmented reality display method for laparoscopic surgery provided in another embodiment of this application;

[0069] Figure 5 A schematic diagram of the structure of the augmented reality display device for laparoscopic surgery provided in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0071] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] The terms “first,” “second,” etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, products, or apparatus.

[0074] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0075] In related technologies, intraoperative endoscopic images and preoperative 3D models of the surgical site are typically registered based on feature points or contours of the site, thereby achieving augmented reality display of the site under laparoscopic vision. This method has been successfully applied in surgeries on sites such as the liver and uterus, mainly relying on anatomical landmarks such as abundant vascular branches and clear organ boundaries for registration. For example, in liver surgery, the vascular network of the hepatic artery, portal vein, and hepatic vein can be used as registration landmarks; in uterine surgery, anatomical structures such as the uterine artery and fallopian tubes can be used for localization. However, for some sites with fewer anatomical landmarks, such as the kidney, the kidney is located retroperitoneally, and the anatomical structures visible intraoperatively are extremely limited. A large amount of adipose tissue surrounds the kidney, obscuring most of the basic anatomical structures. The main visible features are only the surface of the renal parenchyma and the outline of exophytic tumors. Unlike liver and uterine surgeries, it is impossible to use the rich vascular network for registration, and there is a lack of stable and reliable anatomical landmarks for registration constraints.

[0076] Manual registration methods suffer from large error ranges and low efficiency. Automatic registration methods perform poorly in complex anatomical structures, lack robust registration algorithms, and are easily affected by factors such as image quality and lighting conditions. Their registration accuracy cannot meet the requirements for precise tumor localization in procedures such as partial nephrectomy. Patient-specific training methods require individual model training for each patient to adapt to their anatomical features. However, synthetic data and real surgical images have significant domain differences, making a universal solution applicable to multiple patients with a single training iteration difficult for clinical implementation. Location-based feature point registration methods are ineffective in location-specific surgeries due to the sparse and unstable surface features of locations (such as the kidney). Contour-based registration methods are limited by the simplicity and similarity of location contours, resulting in limited registration accuracy and an inability to handle partially occluded or deformed locations. Furthermore, some existing neural convolutional network methods first segment the surgical region using a neural convolutional network and estimate its position and proportion, then predict 3D pose parameters using a rotational regression network. These methods rely on a large amount of high-quality training data and are prone to failure when the surgical site moves or deforms. They require multiple stable endoscopic images to obtain stable results, lack reliable failure recovery mechanisms, and cannot meet the real-time requirements of surgery, thus affecting the surgical procedure. These shortcomings severely restrict the clinical application and promotion of augmented reality technology in surgeries on sites with few anatomical landmarks (such as the kidneys).

[0077] To address the aforementioned technical issues, the augmented reality display method for laparoscopic surgery provided in this application introduces a standard statistical shape model constructed based on collective anatomical knowledge as a universal shape prior. This model unifies the specific 3D part models of different objects into a single standard space through registration, eliminating the specificity of target parts for different objects. This allows for rendering and overlaying of the current frame image without relying on anatomical landmarks on the target part's surface; instead, it constrains the shape of the target 3D part model through collective anatomical knowledge, significantly improving registration accuracy and thus enhancing the precision of the augmented reality display. Furthermore, by utilizing the complex nonlinear mapping relationship from endoscopic images to camera pose in the standard statistical shape model space learned by an end-to-end neural network, the intraoperative endoscopic images are processed, and the estimated camera pose parameters corresponding to the current frame image contained within the endoscopic image are output in real time. This achieves a direct mapping from the 3D part model to the endoscopic image, effectively avoiding registration failures caused by reliance on local and variable anatomical landmarks, thus improving real-time performance and accuracy.

[0078] The application scenarios of the embodiments of this application will be described below first.

[0079] The augmented reality display method for laparoscopic surgery provided in this application is not limited to augmented reality display of the current frame image of the target area in laparoscopic surgery.

[0080] Figure 1 A scene illustration of the augmented reality display method for laparoscopic surgery provided in the embodiments of this application, such as... Figure 1 As shown, the specific application scenarios of this application include a computing storage device 11, an image acquisition device 12, and an augmented reality display device 13. For example, the computing storage device is used to train a standard statistical shape model, a preoperative 3D site model, and a projection pose estimation model; the image acquisition device is used to acquire endoscopic images in real time, including the current frame image; the augmented reality display device is used to obtain the current frame image from the image acquisition device, obtain the preoperative 3D site model and projection pose estimation model from the computing storage device 11, and execute the augmented reality display method for laparoscopic surgery provided in this application embodiment, thereby rendering the preoperative 3D site model of the target site into the current frame image under the laparoscopic field of view during the surgery.

[0081] It should be noted that the computing storage device and the augmented reality display device can be the same device or different devices, and there is no limit to the number of devices. Figure 1 This is merely an example and does not constitute a limitation on the implementation of the augmented reality display method for laparoscopic surgery provided in the embodiments of this application.

[0082] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0083] Figure 2 This is a schematic flowchart of the augmented reality display method for laparoscopic surgery provided in an embodiment of this application, as shown below. Figure 2 As shown, the augmented reality display method for this laparoscopic surgery includes:

[0084] S201. Obtain the current frame image of the target area during the surgical procedure.

[0085] The target site is a part of the patient's body with few or no anatomical landmarks, such as the kidneys or spleen. The patient may be a human or an animal.

[0086] For example, an endoscopic image containing a current frame image is received from an image acquisition device, which is typically a two-dimensional image, such as an endoscopic camera system.

[0087] For example, a video stream is acquired in real time from an endoscopic camera system, and the current frame image is extracted from the video stream buffer using built-in software.

[0088] S202. Input the current frame image into the pre-trained projection pose estimation model to obtain the camera pose estimation parameters corresponding to the current frame image. The camera pose estimation parameters represent the position of the standard statistical shape model corresponding to the target part in the camera coordinate system corresponding to the current frame image. The standard statistical shape model is used to capture the shape statistical regularity of the target part and does not depend on the anatomical landmarks of the target part.

[0089] The shapes of different target sites vary considerably, and each target site corresponds to a different standard statistical shape model. For example, the kidney and spleen each correspond to different standard statistical shape models. A standard statistical shape model is a mathematical model obtained by statistically analyzing the three-dimensional models of normal and abnormal target sites (such as the kidney) in a population (e.g., hundreds of cases) using principal component analysis. This standard statistical shape model includes an average shape and a set of principal components. Each principal component represents a major variation model of the target site's shape. By adjusting the corresponding shape parameters, for example, a series of statistically reasonable three-dimensional kidney shapes can be generated.

[0090] In this step, the projection pose estimation model is a trained deep neural network model.

[0091] For example, this projection pose estimation model utilizes a Convolutional Neural Network (CNN) or Transformer backbone network for feature extraction. It extracts high-level, shape-related semantic features from the input current frame image and maps the extracted feature vectors to camera pose estimation parameters through a regression head composed of fully connected layers. These camera pose estimation parameters characterize the position of the standard statistical shape model corresponding to the target region in the camera coordinate system corresponding to the current frame image.

[0092] It should be understood that the projection pose estimation model makes predictions relative to a general standard statistical shape model, rather than a three-dimensional part model of a specific surgical object. Therefore, this projection pose estimation model only needs to be trained once and can be applied to any new surgical object, solving the problem that traditional methods require customized modeling or training for each surgical object, and greatly improving the clinical application scope and deployment efficiency of the system.

[0093] S203. Based on the camera pose estimation parameters, render and overlay the pre-constructed target 3D part model corresponding to the target part onto the current frame image to generate the augmented reality display result of the target part.

[0094] In this step, the camera pose estimation parameters can actually be understood as the specific projection position of the preoperatively constructed target 3D part model in the current frame image, including translation and rotation components.

[0095] The target 3D part model is a specific 3D part model registered to the standard statistical shape model space. For example, from the preoperative medical images of the surgical subject (patient), such as computed tomography (CT) images or magnetic resonance imaging (MRI) images, a 3D part model is segmented and extracted. Then, through a registration algorithm (specifically, a two-stage registration algorithm), a set of optimal shape parameters and transformations are found so that the 3D part model of the surgical subject can be accurately represented by the standard statistical shape model, thus obtaining the above-mentioned target 3D part model. This target 3D part model retains the specificity of the target part of the surgical subject and is placed in a unified standard statistical shape model coordinate system.

[0096] For example, using a graphics rendering engine, a target 3D part model registered to a standard statistical shape model space is placed into a virtual scene, and the target 3D part model is rendered based on camera pose estimation parameters. For example, to avoid obstructing the real field of view, it can be rendered as a semi-transparent colored surface or a clear grid wireframe. The rendered virtual model image is then alpha-blended with the real current frame image to generate the final augmented reality display result.

[0097] In this embodiment, a standard statistical shape model constructed based on group anatomical knowledge is introduced as a universal shape prior. This model can capture the statistical regularity and variation pattern of the shape of a part. Based on this, a target 3D part model that has been registered to the standard statistical shape model space is used. That is, the specific 3D part models of different objects are unified into the same standard space through registration, eliminating the specificity of the target part of different objects. This allows the rendering and overlay of the current frame image to be performed without relying on the anatomical landmarks on the surface of the target part. Instead, the shape of the target 3D part model is constrained by group anatomical knowledge, achieving alignment between the preoperative target 3D part model and the current frame image during surgery. This significantly improves the registration accuracy, thereby improving the accuracy of augmented reality display, providing doctors with stable augmented reality navigation, reducing the risk of intraoperative errors, and increasing the success rate of surgery. Furthermore, by leveraging the complex nonlinear mapping relationship learned by the end-to-end neural network from endoscopic images to the camera pose in the standard statistical shape model space, the intraoperative endoscopic images are processed, and the camera pose estimation parameters corresponding to the current frame image contained in the endoscopic image are output in real time. These camera pose estimation parameters, constrained by the global shape prior of the standard statistical shape model, conform to anatomical laws and are used to accurately back-map the preoperative target 3D model to the target position in the current frame image, achieving a direct mapping from the 3D model to the endoscopic image. This effectively avoids registration failures caused by dependence on local and variable anatomical landmarks and eliminates the time-consuming iterative optimization steps in traditional registration, improving real-time performance and accuracy. In addition, since the projection pose estimation model uses the standard statistical shape model, which is universal and does not depend on specific individual anatomical landmarks, as a unified geometric reference system, this projection pose estimation model is applicable to any surgical object that conforms to this statistical law, solving the problem in related technologies that require customized training models for each specific object, and greatly improving the system's versatility.

[0098] The following provides a detailed description of the construction method of the standard statistical shape model provided in the embodiments of this application.

[0099] In some embodiments, the standard statistical shape model is constructed as follows:

[0100] Step 1.1: Obtain multiple sample medical images of the target area, including medical images of the target area when it is normal and medical images of the target area when it is abnormal.

[0101] The sample medical images can be considered as the raw data used to construct the standard statistical shape model. For example, they can be medical images of target sites of different subjects (patients) or the same subject at different times (such as childhood, adulthood, and old age). To ensure the universality of the constructed standard statistical shape model, the sample medical images can simultaneously include images of normal (such as healthy target sites) and abnormal (such as target sites with diseases such as cysts or tumors).

[0102] For example, the sample medical images are CT and / or MRI images.

[0103] Step 1.2: Extract the three-dimensional contour of each sample medical image from multiple sample medical images to obtain the first three-dimensional part model corresponding to each sample medical image.

[0104] For example, using deep learning segmentation algorithms (such as U-Net), interactive segmentation software, or a combination of expert annotation and automatic segmentation, the boundary of the target area is delineated in each sample medical image, distinguishing the tissue of the target area from the surrounding fat and blood vessels. Then, the segmentation results are stacked, and algorithms such as moving cubes are used to generate a three-dimensional mesh model of the target area surface composed of triangular facets, i.e., the first three-dimensional area model.

[0105] It should be noted that this embodiment does not impose specific limitations on the implementation method of obtaining the first three-dimensional part model, and can refer to existing technologies.

[0106] Step 1.3: Based on the least squares method, iteratively align multiple first three-dimensional part models to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape.

[0107] For example, the vertex coordinates of multiple first 3D part models are averaged to obtain an initial and coarse average shape of the target part. For each first 3D part model, an optimal rigid body transformation is calculated such that after the transformation, the sum of squared distances between all vertices of the first 3D part model and the corresponding vertices of the current average shape is minimized. Then, based on the newly aligned multiple first 3D part models, the average shape of the target part is recalculated. The above process of alignment and calculation of the new average shape is repeated until the change in the average shape is less than a preset threshold, that is, the least squares method converges. At this point, a stable and more representative average shape and aligned multiple second 3D part models are obtained.

[0108] This step can be summarized by the following calculation process:

[0109] 1.3.1 Calculate the average shape.

[0110]

[0111] In the formula, For the first The average shape of the target region in each iteration; This represents the number of models for the first three-dimensional part. For the first The first three-dimensional part model in the first The shape at the next iteration.

[0112] 1.3.2 Align each first 3D part model to the average shape.

[0113]

[0114] In the formula, This represents the total number of vertices in the first three-dimensional part model. For vertex index, Let be the transformation matrix.

[0115] 1.3.3 Repeat steps 1.3.1 to 1.3.2 until the change in the average shape is less than the preset threshold, i.e., the least squares method converges.

[0116] Step 1.4: Based on the average shape and multiple second three-dimensional part models, perform principal component analysis of the shape to obtain the standard statistical shape model corresponding to the target part.

[0117] Shape principal component analysis (PCA) involves identifying the most dominant shape variation patterns, or principal components, from aligned shape data (multiple second-dimensional part models), thereby achieving data dimensionality reduction. Each principal component represents a direction of shape variation, such as size or aspect ratio, and the eigenvalues ​​of the principal components characterize the importance of that variation pattern in the population.

[0118] For example, this step can be achieved through the following calculation process:

[0119] 1.4.1 Calculate the covariance matrix.

[0120]

[0121] In the formula, For the first A second three-dimensional part model The average shape of the target region; N is the number of second 3D region models; The covariance matrix is ​​defined by the elements on the diagonal, which represent the variance of each vertex's coordinates, and the elements off the diagonal, which represent the covariance between different vertex coordinates.

[0122] 1.4.2 Eigenvalue decomposition.

[0123]

[0124] In the formula, , represents the eigenvector matrix, where each eigenvector represents a principal component; , represents the eigenvalue diagonal matrix, where the magnitude of each eigenvalue quantifies the importance of the mutation pattern represented by its corresponding eigenvector.

[0125] For example, the first principal component (i.e. the largest eigenvalue) represents the size variation pattern of the overall size of the target part, the second principal component (i.e. the second largest eigenvalue) represents the variation pattern of the aspect ratio of the target part, and the third principal component may represent the concavity and convexity variation pattern at a specific location of the target part.

[0126] 1.4.3 Principal component selection and standard statistical shape model generation.

[0127] Choose the previous one Principal components (e.g., the first 20) are used to explain, for example, more than 95% of the shape variation. The final standard statistical shape model is defined by the average shape, the selected principal component matrix, and the corresponding eigenvalues.

[0128]

[0129] In the formula, These are the eigenvalues ​​arranged in descending order in the eigenvalue diagonal matrix. Indicates the order of eigenvalues. This indicates the order in which eigenvalues ​​are assigned values ​​in order to retain 95% of the explanatory power.

[0130] After principal component selection is completed, the shape of any statistically reasonable target region is determined. All can be generated or represented in the following ways:

[0131]

[0132] In the formula, For the first The shape parameters follow a normal distribution. Restricted to Within the range; Indicates the first Each feature value.

[0133] In addition, the coordinate system of the standard statistical shape model is defined with the origin as the centroid of the target part, the Z-axis along the largest eigenvector of the inertial tensor (e.g., the major axis of the kidney), the X-axis through the renal hilum, and the Y-axis determined by the right-hand rule.

[0134] The standard statistical shape model constructed in this application embodiment can be used as prior knowledge to guide and constrain the registration results, achieving registration without relying on visible anatomical landmarks and improving registration accuracy.

[0135] To improve the accuracy of the average shape and standard statistical shape models, one possible implementation involves iteratively aligning multiple first three-dimensional part models based on the least squares method to obtain the average shape of the target part and multiple second three-dimensional part models aligned with the average shape. Specifically, this includes: preprocessing the multiple first three-dimensional part models to obtain preprocessed first three-dimensional part models, where preprocessing includes mesh resampling, topology repair, and surface smoothing; and iteratively aligning the preprocessed first three-dimensional part models based on the least squares method to obtain the average shape of the target part and multiple second three-dimensional part models aligned with the average shape.

[0136] Mesh resampling unifies the meshes of multiple first 3D part models to a standard structure with the same number of vertices and the same mesh connection structure, ensuring the feasibility of vertex-by-vertex mathematical comparison and calculation between different first 3D part models; topology repair is used to repair possible errors in the mesh, such as holes and non-manifold edges, ensuring that each first 3D part model is a complete and closed surface, ensuring the geometric validity of the model; surface smoothing is used to remove surface irregularities and burrs caused by image noise or segmentation errors, making the model surface smoother and closer to the real anatomical structure.

[0137] After preprocessing multiple first three-dimensional part models, multiple preprocessed first three-dimensional part models with uniform structure and controllable quality are obtained. Then, iterative alignment processing is performed to generate the average shape of the target part.

[0138] Next, combined Figure 3 The construction method of the above standard statistical shape model is summarized and explained. Figure 3 This is a flowchart illustrating the construction method of the standard statistical shape model provided in the embodiments of this application. Figure 3As shown, the entire construction process involves acquiring training data for the standard statistical shape model, data preprocessing, and the specific construction of the model. Taking the construction of the standard statistical shape model of the kidney as an example, multiple CT and / or MRI medical images (e.g., 100, 200, 500, etc.) encompassing normal and various abnormal kidneys are collected to form the training data for the standard statistical shape model. For example, through a combination of expert annotation and automatic segmentation, the corresponding three-dimensional contours of the kidneys are extracted from multiple medical images. Then, each three-dimensional contour of the kidney is subjected to mesh resampling, topology repair, and surface smoothing, thereby unifying multiple three-dimensional kidney models to a uniform mesh density. Based on the multiple three-dimensional kidney models obtained through preprocessing, average shape calculation and principal component analysis are performed sequentially to finally obtain the standard statistical shape model of the kidney.

[0139] The training process of the projection pose estimation model will be explained in detail below.

[0140] First, we will explain the training dataset used during the training of the projection pose estimation model.

[0141] In some embodiments, the training dataset used during training of the projection pose estimation model is obtained in the following way:

[0142] Step 2.1: Obtain multiple keyframe sample images from the sample video stream of the target area during the surgical process. The keyframe sample images cover different viewpoints and different surgical stages of the target area.

[0143] Keyframe sample images are, for example, representative static frame sample images selected by experts from a continuous surgical sample video stream. The keyframe sample images need to cover different stages of the surgical process (such as exploration, separation, cutting, and suturing) and different perspectives of the target area (such as front, side, long view, and close-up) to ensure the diversity and comprehensiveness of the data.

[0144] For example, complete video streams of 50 laparoscopic kidney surgeries are collected, and experts select, for example, 100 keyframes from each surgical video stream, ultimately resulting in 5,000 keyframe sample images.

[0145] Step 2.2: For each keyframe sample image in the multiple keyframe sample images, perform contour fitting between the keyframe sample image and the third three-dimensional part model constructed before the operation of the target part to obtain the first spatial transformation matrix of the third three-dimensional part model in the camera coordinate system corresponding to the keyframe sample image. The third three-dimensional part model is a three-dimensional part model registered to the standard statistical shape model space.

[0146] In this step, contour fitting involves translating, rotating, and scaling the 3D part model to make the projected contour of the 3D part model coincide as accurately as possible with the real contour of the target part in the 2D image (keyframe sample image).

[0147] For example, using specialized annotation software, the third 3D part model is rotated, translated, and scaled to align its projected contour under the virtual camera with the boundary of the actual target part in the keyframe sample image. When the alignment effect is optimal, the specialized annotation software records the transformation matrix required at this point, i.e., the first spatial transformation matrix. When there are 5000 keyframe sample images, the above contour fitting operation needs to be performed on each of these 5000 keyframe sample images to obtain the first spatial transformation matrix corresponding to each of the 5000 keyframe sample images.

[0148] It should be noted that the third-dimensional site model and the keyframe sample image are for the same surgical object (patient). If the keyframe sample image comes from different surgical objects (patients), then a preoperative third-dimensional site model needs to be constructed for each surgical object.

[0149] Step 2.3: Based on the first spatial transformation matrix and the pre-constructed second spatial transformation matrix, determine the camera attitude parameters of the standard statistical shape model in the camera coordinate system. The second spatial transformation matrix represents the spatial projection position transformation relationship from the fourth three-dimensional part model to the standard statistical shape model. The fourth three-dimensional part model is a three-dimensional part model that has not been registered to the standard statistical shape model space.

[0150] It should be understood that registering the fourth 3D part model to the standard statistical shape model space yields the third 3D part model. The fourth 3D part model can be obtained by extracting the 3D contour of the target sample from preoperative CT or MRI images and performing preprocessing (mesh resampling, topology repair, and surface smoothing).

[0151] For example, the first spatial transformation matrix is ​​denoted as Let represent the first spatial transformation matrix corresponding to the e-th keyframe sample image; the second spatial transformation matrix is ​​denoted as . The camera attitude parameters are denoted as... ,but .

[0152] Step 2.4: Normalize the camera intrinsic parameters of multiple keyframe sample images into standard intrinsic parameters.

[0153] Considering the potential differences in camera parameters between endoscopic camera systems used in different surgical procedures, frame images of the same patient acquired by different cameras may exhibit imaging variations. To eliminate these imaging differences caused by the hardware parameters of different cameras, for example, based on a predefined standard intrinsic parameter matrix (e.g., corresponding to a virtual camera with a suitable focal length, no distortion, and the principal point centered in the image), the camera intrinsic parameters corresponding to the keyframe sample images are normalized to obtain the standard intrinsic parameters for the keyframe sample images. For instance, normalizing the camera intrinsic parameter elements to a dimensionless standard form eliminates the scale differences in focal length and principal point at different resolutions for different cameras, making the projection pose estimation model more consistent in input dimension and imaging scale, and improving cross-device (here referring to camera systems) generalization.

[0154] Step 2.5: Integrate the keyframe sample images, standard intrinsic parameters, and camera pose parameters to form a training dataset.

[0155] For example, 5000 keyframe sample images, standard intrinsic parameters, and camera pose parameters are stored according to a preset format to form a training dataset:

[0156]

[0157] In the formula, This represents the training dataset; This represents the standard intrinsic parameter corresponding to the e-th keyframe sample image; This represents the camera pose parameters corresponding to the e-th keyframe sample image; This represents the e-th keyframe sample image.

[0158] It should be understood that by using this training dataset to train the projection pose estimation model, the projection pose estimation model learns the relationship between real endoscopic frame images and camera pose parameters during surgery.

[0159] Optionally, between adjacent keyframes, interpolation combined with expert manual correction can be used to obtain the first spatial transformation matrix over the entire video stream time frame, thereby obtaining the camera pose parameters over the entire video stream time frame, ultimately achieving the goal of expanding the training dataset.

[0160] In this embodiment, the training dataset teaches the neural network to learn camera pose parameters related to a standard statistical shape model, rather than camera pose parameters related to a specific patient's 3D body part model. This enables the trained projection pose estimation model to generalize across patients, which is key to achieving "train once, apply to multiple patients." By normalizing camera intrinsic parameters, the imaging differences caused by different camera hardware parameters are eliminated. This allows the neural network to focus on learning the essential mapping relationship between frame images and camera pose parameters without needing to learn redundant lens distortion compensation, thus improving training efficiency and model performance.

[0161] For example, in some possible implementations, the second space transformation matrix is ​​obtained as follows:

[0162] Step 3.1: Apply the least squares method to perform coarse registration between the fourth three-dimensional part model and the standard statistical shape model based on the centroid position and principal axis position of the standard statistical shape model, so as to obtain the coarse registration spatial transformation matrix and the coarse registration fourth three-dimensional part model.

[0163] This step can be considered the first stage of registration, used to resolve large-scale, global differences in position, orientation, and scale between two 3D models.

[0164] For example, the centroids of the standard statistical shape model and the fourth 3D part model are calculated. A translation transformation is then used to move the centroid of the fourth 3D part model to the centroid of the standard statistical shape model. Next, the fourth 3D part model is calculated and rotated so that its major axis aligns with the major axis of the standard statistical shape model, and its minor axis aligns with the corresponding axis of the standard statistical shape model. This composite transformation of translation followed by rotation is the coarse registration spatial transformation matrix. Applying this matrix to the fourth 3D part model yields a coarsely registered fourth 3D part model, which is now roughly aligned with the standard statistical shape model.

[0165] Step 3.2: Apply the least squares method, based on deformation estimation and iterative optimization, to perform fine registration on the coarsely registered fourth three-dimensional part model and the standard statistical shape model, and obtain the fine registration spatial transformation matrix.

[0166] This step can be regarded as the second stage of registration. Based on the results of coarse registration, fine and local pose adjustments are made to ultimately achieve the highest accuracy alignment between the patient model and the standard statistical shape model.

[0167] This step can be summarized by the following calculation process:

[0168] 3.2.1 Shape parameter estimation.

[0169] The deviation between the coarsely registered fourth 3D part model and the standard statistical shape model is projected onto the principal components of the standard statistical shape to obtain a set of initial shape parameters. .

[0170]

[0171] In the formula, This is the fourth 3D part model after coarse registration; Represents the standard statistical shape model. Indicates principal component, Indicates the first step in the precision matching process Next iteration index.

[0172] 3.2.2. Standard statistical shape model instance generation.

[0173] Using the shape parameters in 3.2.1, a personalized standard statistical shape model instance can be generated using the following formula.

[0174]

[0175] 3.2.3 Calculation of the spatial transformation matrix for precise registration.

[0176] Solving for a fine rigid body transformation (including small rotations and translations) that minimizes the sum of squared vertex distances between the transformed fourth 3D part model and the generated standard statistical shape model instance can be represented by the following formula:

[0177]

[0178] In the formula, This represents the total number of vertices in the fourth three-dimensional part model. For vertex index, This represents the space transformation matrix for fine registration during the iterative process.

[0179] 3.2.4 Repeat steps 3.2.1 to 3.2.3 until the changes in shape parameters and registration error are less than the preset threshold, at which point the algorithm converges.

[0180] Repeat steps 3.2.1 to 3.2.3 to continuously optimize the shape parameters. And the fine registration space transformation matrix The algorithm converges when the changes in both values ​​are less than the corresponding preset thresholds.

[0181] Step 3.3: The product of the coarse registration spatial transformation matrix and the fine registration spatial transformation matrix is ​​determined as the second spatial transformation matrix.

[0182] The second spatial transformation matrix contains all the registration information from coarse to fine.

[0183] The process of obtaining the second spatial transformation matrix described above can be regarded as the process of projecting the three-dimensional part model of the target part of the surgical object (patient) onto the standard statistical shape model space.

[0184] It should be noted that in other embodiments of this application, the registration of the target site's three-dimensional site model to the standard statistical shape model space before surgery (e.g., the construction of the target three-dimensional site model in step S203) can all refer to the implementation process of steps 3.1 to 3.3.

[0185] In this embodiment, a two-stage strategy of coarse-to-fine registration is adopted. The coarse registration provides a reliable initial estimate, effectively preventing the fine registration algorithm from getting trapped in erroneous local optima due to poor initial positioning, thus ensuring the robustness and convergence of the registration process. The fine registration stage achieves high-precision spatial mapping through iterative optimization and deformation estimation. Furthermore, the complex and personalized registration problem is transformed into a unified matrix, enabling the use of a general neural network (i.e., a projection pose estimation model) to estimate camera pose during surgery, thereby achieving both real-time performance and versatility of the system.

[0186] Secondly, in some embodiments, the loss function of the projection pose estimation model includes a pose estimation loss function and a shape loss function.

[0187] The pose estimation loss function directly guides the model to learn the correct camera pose estimation parameters by calculating the difference between the camera pose estimation parameters predicted by the projection pose estimation model and the real camera pose labels in the training dataset. The shape loss function is calculated from the backprojection results of the preoperative 3D body part model and the surface error of the hand-labeled real labels. In other words, the shape loss function does not directly compare the camera pose values, but rather compares the result of backprojecting the preoperative 3D body part model using the predicted camera pose estimation parameters onto the intraoperative endoscopic image, with the result of backprojecting the preoperative 3D body part model using the hand-labeled real camera pose labels in the training dataset onto the intraoperative endoscopic image. This ensures that the camera pose estimation parameters predicted by the projection pose estimation model are geometrically accurate in 3D space.

[0188] The core of training the projection pose estimation model is to use the training dataset constructed above to supervise the training of the projection pose estimation model to be trained, and to optimize it by combining loss functions, so as to obtain a well-trained projection pose estimation model.

[0189] For example, the total loss function of the projected pose estimation model can be expressed as:

[0190]

[0191] In the formula, This represents the attitude estimation loss function; Represents the shape loss function; and These are hyperparameters used to adjust the weights of the pose estimation loss function and the shape loss function in the total loss function.

[0192] In summary, the projection pose estimation model, during training, jointly optimizes the pose estimation loss and shape loss, ensuring that the output camera pose estimation parameters are constrained by global shape priors, thus guaranteeing the accuracy of the results and achieving a balance between real-time performance and accuracy. Simultaneously, the iterative error feedback mechanism continuously corrects the camera pose estimation parameter errors through backpropagation, enhancing the projection pose estimation model's adaptability to dynamic changes and its robustness under target movement or image quality fluctuations.

[0193] In some embodiments, obtaining the current frame image of the target area during the operation includes: obtaining the original image of the target area during the operation; applying a preset standard intrinsic parameter matrix to perform camera intrinsic parameter normalization processing on the original image to obtain the current frame image of the target area during the operation.

[0194] The raw image can be considered as an unprocessed frame image directly acquired from the image acquisition device. This raw image may contain inherent distortions of the camera lens (such as barrel distortion and pincushion distortion), and its imaging geometry is determined by the camera's specific intrinsic parameter matrix. The standard intrinsic parameter matrix is ​​a preset, relatively idealized camera intrinsic parameter matrix that defines a virtual standard camera.

[0195] For example, receiving raw images from an image acquisition device, such as an endoscope camera system.

[0196] For example, a video stream is acquired in real time from an endoscopic camera system, and the raw image of the current frame is extracted from the video stream buffer using built-in software.

[0197] Furthermore, based on the standard intrinsic parameter matrix and the intrinsic parameter matrix corresponding to the camera that acquired the original image (understandably, the intrinsic parameter matrix of the camera used to acquire the original image of the target area during surgery is known), the original image is transformed to make it geometrically equivalent to the image captured by the aforementioned standard camera, i.e., the current frame image of the target area during the surgical process. This process eliminates intrinsic parameter differences and lens distortion between different cameras.

[0198] In this embodiment, by performing camera intrinsic parameter normalization on the original image, the images input to the projection pose estimation model all have uniform and known imaging geometric parameters, eliminating systematic errors caused by different camera hardware, ensuring strict uniformity of camera intrinsic parameters during training and inference, thereby guaranteeing accurate alignment of the final augmented reality overlay result without misalignment.

[0199] Next, combined Figure 4 The augmented reality display method for laparoscopic surgery provided in the embodiments of this application will be further described.

[0200] Figure 4 A schematic flowchart of an augmented reality display method for laparoscopic surgery provided in another embodiment of this application is shown below. Figure 4 As shown, the augmented reality display methods for laparoscopic surgery include:

[0201] S401. Obtain the original image of the target area during the surgical procedure.

[0202] S402. Perform camera intrinsic parameter normalization on the original image.

[0203] By applying a preset standard intrinsic parameter matrix, the original image is processed to perform camera intrinsic parameter normalization to obtain the current frame image of the target area during the surgical procedure.

[0204] S403. Input the current frame image obtained after normalizing the camera intrinsic parameters into the pre-trained projection pose estimation model.

[0205] The projection pose estimation model performs feature extraction and pose estimation on the current frame image and outputs camera pose estimation parameters.

[0206] S404. Based on the camera pose estimation parameters, backproject the pre-built target 3D part model corresponding to the target part into the current frame image to generate the augmented reality display result of the target part.

[0207] The target three-dimensional part model is a specific three-dimensional part model that has been registered to the standard statistical shape model space. For example, a three-dimensional part model is segmented and extracted from the preoperative medical images of the surgical object (patient), such as CT images or MRI images. Then, through the registration algorithm in the aforementioned embodiment (see the implementation process of steps 3.1 to 3.3 for details), a set of optimal shape parameters and transformations are found so that the three-dimensional part model of the surgical object can be accurately represented by the standard statistical shape model, thereby obtaining the above-mentioned target three-dimensional part model. This target three-dimensional part model retains the specificity of the target part of the surgical object and is placed in a unified standard statistical shape model coordinate system.

[0208] For example, using a graphics rendering engine, a target 3D part model registered to a standard statistical shape model space is placed into a virtual scene, and the target 3D part model is rendered based on camera pose estimation parameters. For example, to avoid obstructing the real field of view, it can be rendered as a semi-transparent colored surface or a clear grid wireframe. The rendered virtual model image is then alpha-blended with the real current frame image to generate the final augmented reality display result.

[0209] In summary, this application realizes augmented reality display technology for target areas lacking visible anatomical markers during laparoscopic surgery. Based on a standard statistical shape model, it can adaptively adapt to various anatomical variations and pathological conditions of target areas (such as the kidney). The end-to-end neural network registration scheme significantly reduces the need for manual intervention during surgery and effectively improves the accuracy and stability of real-time projection display during surgery.

[0210] Furthermore, a universal registration system applicable to multiple patients with a single training iteration is achieved through a standard statistical shape model. This system innovatively integrates shape parameter regression and pose estimation into a single neural network, enabling end-to-end learning from endoscopic images to registration parameters (i.e., camera pose estimation parameters). Combined with shape constraints and an iterative error feedback mechanism, this significantly improves registration accuracy and robustness, meeting the real-time requirements of surgery.

[0211] Figure 5 This is a schematic diagram of the structure of the augmented reality display device for laparoscopic surgery provided in the embodiments of this application, as shown below. Figure 5 As shown, the augmented reality display device 50 for laparoscopic surgery provided in this embodiment includes: an acquisition module 51, an input module 52, and a display enhancement module 53. Wherein:

[0212] The acquisition module 51 is used to acquire the current frame image of the target area during the surgical process;

[0213] Input module 52 is used to input the current frame image into the pre-trained projection pose estimation model to obtain the camera pose estimation parameters corresponding to the current frame image. The camera pose estimation parameters represent the position of the standard statistical shape model corresponding to the target part in the camera coordinate system corresponding to the current frame image. The standard statistical shape model is used to capture the shape statistical regularity of the target part and does not depend on the anatomical landmarks of the target part.

[0214] The display enhancement module 53 is used to render and overlay the pre-built target three-dimensional part model corresponding to the target part onto the current frame image based on the camera pose estimation parameters, so as to generate the augmented reality display result of the target part. The target three-dimensional part model is a specific three-dimensional part model that has been registered to the standard statistical shape model space.

[0215] In one possible implementation, the standard statistical shape model is constructed as follows:

[0216] Acquire multiple sample medical images of the target area, including medical images of the target area when it is normal and medical images of the target area when it is abnormal;

[0217] The three-dimensional contour of each sample medical image is extracted from multiple sample medical images to obtain the first three-dimensional part model corresponding to each sample medical image;

[0218] Based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape.

[0219] Based on the average shape and multiple second-dimensional part models, principal component analysis of shape is performed to obtain the standard statistical shape model corresponding to the target part.

[0220] In one possible implementation, based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape, including:

[0221] Multiple first three-dimensional part models are preprocessed to obtain multiple preprocessed first three-dimensional part models. The preprocessing includes mesh resampling, topology repair and surface smoothing.

[0222] Based on the least squares method, the preprocessed first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and the multiple second three-dimensional part models aligned with the average shape.

[0223] In one possible implementation, the training dataset used during training of the projection pose estimation model is obtained as follows:

[0224] The keyframe sample images are obtained from the sample video stream of the target area during the operation. The keyframe sample images cover different viewpoints and different stages of the operation.

[0225] For each keyframe sample image in multiple keyframe sample images, the keyframe sample image is contour-fitted with the third three-dimensional part model of the target part constructed before surgery to obtain the first spatial transformation matrix of the third three-dimensional part model in the camera coordinate system corresponding to the keyframe sample image. The third three-dimensional part model is a three-dimensional part model registered to the standard statistical shape model space.

[0226] Based on the first spatial transformation matrix and the pre-constructed second spatial transformation matrix, the camera pose parameters of the standard statistical shape model in the camera coordinate system are determined. The second spatial transformation matrix represents the spatial projection position transformation relationship from the fourth three-dimensional part model to the standard statistical shape model. The fourth three-dimensional part model is a three-dimensional part model that has not been registered to the standard statistical shape model space.

[0227] The camera intrinsic parameters of multiple keyframe sample images are normalized to standard intrinsic parameters;

[0228] The keyframe sample images, standard intrinsic parameters, and camera pose parameters are integrated to form a training dataset.

[0229] In one possible implementation, the second spatial transformation matrix is ​​obtained as follows:

[0230] By applying the least squares method, based on the centroid and principal axis positions of the standard statistical shape model, coarse registration is performed between the fourth three-dimensional part model and the standard statistical shape model to obtain the coarse registration spatial transformation matrix and the coarse registration fourth three-dimensional part model.

[0231] By applying the least squares method, based on deformation estimation and iterative optimization, fine registration is performed on the coarsely registered fourth three-dimensional part model and the standard statistical shape model to obtain the fine registration spatial transformation matrix.

[0232] The product of the coarse registration spatial transformation matrix and the fine registration spatial transformation matrix is ​​determined as the second spatial transformation matrix.

[0233] In one possible implementation, the loss function of the projection pose estimation model includes a pose estimation loss function and a shape loss function.

[0234] In one possible implementation, acquiring the current frame image of the target site during the surgical procedure includes:

[0235] Obtain raw images of the target area during the surgical procedure;

[0236] By applying a preset standard intrinsic parameter matrix, the original image is processed to perform camera intrinsic parameter normalization to obtain the current frame image of the target area during the surgical procedure.

[0237] The augmented reality display device for laparoscopic surgery provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0238] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6As shown, the augmented reality display device 13 (i.e., electronic device) provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the augmented reality display device 13 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0239] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0240] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0241] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0242] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0243] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0244] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0245] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0246] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0247] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0248] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0249] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0250] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0251] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0252] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0253] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An augmented reality display method for laparoscopic surgery, characterized in that, include: Acquire the current frame image of the target area during the surgical procedure; The current frame image is input into a pre-trained projection pose estimation model to obtain the camera pose estimation parameters corresponding to the current frame image. The camera pose estimation parameters represent the position of the standard statistical shape model corresponding to the target part in the camera coordinate system corresponding to the current frame image. The standard statistical shape model is used to capture the shape statistical regularity of the target part and does not depend on the anatomical landmarks of the target part. Based on the camera pose estimation parameters, the pre-constructed target 3D part model corresponding to the target part is rendered and superimposed onto the current frame image to generate the augmented reality display result of the target part. The target 3D part model is a specific 3D part model registered to the standard statistical shape model space.

2. The augmented reality display method according to claim 1, characterized in that, The standard statistical shape model is constructed in the following manner: Acquire multiple sample medical images of the target area, including medical images of the target area when it is normal and medical images of the target area when it is abnormal; The three-dimensional contour of each of the multiple sample medical images is extracted to obtain the first three-dimensional part model corresponding to each of the multiple sample medical images; Based on the least squares method, multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape. Based on the average shape and the multiple second three-dimensional part models, principal component analysis of shape is performed to obtain the standard statistical shape model corresponding to the target part.

3. The augmented reality display method according to claim 2, characterized in that, The step of iteratively aligning multiple first three-dimensional part models based on the least squares method to obtain the average shape of the target part, and multiple second three-dimensional part models aligned with the average shape, includes: Multiple first three-dimensional part models are preprocessed to obtain multiple preprocessed first three-dimensional part models. The preprocessing includes mesh resampling, topology repair and surface smoothing. Based on the least squares method, the preprocessed multiple first three-dimensional part models are iteratively aligned to obtain the average shape of the target part and multiple second three-dimensional part models aligned with the average shape.

4. The augmented reality display method according to any one of claims 1 to 3, characterized in that, The training dataset used during training of the projection pose estimation model was obtained in the following manner: Multiple keyframe sample images are acquired from a sample video stream of the target area during the surgical procedure, and the keyframe sample images cover different viewpoints and different surgical stages of the target area. For each keyframe sample image in the plurality of keyframe sample images, the keyframe sample image is contour-fitted with the third three-dimensional part model of the target part constructed before the operation to obtain the first spatial transformation matrix of the third three-dimensional part model in the camera coordinate system corresponding to the keyframe sample image. The third three-dimensional part model is a three-dimensional part model registered to the standard statistical shape model space. Based on the first spatial transformation matrix and the pre-constructed second spatial transformation matrix, the camera pose parameters of the standard statistical shape model in the camera coordinate system are determined. The second spatial transformation matrix represents the spatial projection position transformation relationship from the fourth three-dimensional part model to the standard statistical shape model. The fourth three-dimensional part model is a three-dimensional part model that has not been registered to the space of the standard statistical shape model. The camera intrinsic parameters of the multiple keyframe sample images are normalized to standard intrinsic parameters; The keyframe sample images, the standard intrinsic parameters, and the camera pose parameters are integrated to form the training dataset.

5. The augmented reality display method according to claim 4, characterized in that, The second spatial transformation matrix is ​​obtained as follows: Using the least squares method, based on the centroid position and principal axis position of the standard statistical shape model, coarse registration is performed on the fourth three-dimensional part model and the standard statistical shape model to obtain the coarse registration spatial transformation matrix and the coarse registration fourth three-dimensional part model. By applying the least squares method, based on deformation estimation and iterative optimization, fine registration is performed on the coarsely registered fourth three-dimensional part model and the standard statistical shape model to obtain the fine registration spatial transformation matrix. The product of the coarse registration spatial transformation matrix and the fine registration spatial transformation matrix is ​​determined as the second spatial transformation matrix.

6. The augmented reality display method according to any one of claims 1 to 3, characterized in that, The loss function of the projection attitude estimation model includes an attitude estimation loss function and a shape loss function.

7. The augmented reality display method according to any one of claims 1 to 3, characterized in that, The acquisition of the current frame image of the target area during the surgical procedure includes: Obtain raw images of the target area during the surgical procedure; By applying a preset standard intrinsic parameter matrix, the original image is subjected to camera intrinsic parameter normalization processing to obtain the current frame image of the target area during the surgical process.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed, implements the method of any one of claims 1 to 7.

Images