Preoperative and intraoperative organ model holographic fusion method and system based on three-dimensional registration

Abstract

The invention relates to the technical field of medical image processing, in particular to a preoperative and intraoperative organ model holographic fusion method and system based on three-dimensional registration, and the method comprises the following steps: converting a to-be-registered preoperative organ model and a to-be-registered intraoperative organ model from a three-dimensional point cloud into a two-dimensional projection; respectively obtaining an XY plane two-dimensional projection, a YZ plane two-dimensional projection and an XZ plane two-dimensional projection of the preoperative organ model and the intraoperative organ model; constructing a three-dimensional registration model by using a GAN network; obtaining a registration model of the preoperative organ model based on the three-dimensional registration model; the registration model of the preoperative organ model is projected to the preoperative organ model in a holographic mode, so that a preoperative and intraoperative organ model holographic fusion result is obtained. According to the method, registration of the organ models before and during the operation is refined into registration of the multiple two-dimensional images from three-dimensional overall registration, parallel registration processing is carried out on the two-dimensional images through the three registration models, the registration complexity and the information processing amount are reduced, the registration efficiency is improved, and the timeliness requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and system for holographic fusion of preoperative and intraoperative organ models based on three-dimensional registration. Background Art

[0002] During laparoscopic surgery, doctors operating surgical robots use images of organs transmitted to monitors to understand the real-time surgical scene and guide the use of surgical instruments. Although the images returned by the monitor can accurately reflect the condition of the organs being operated on, the limited field of view of the endoscope and obstruction caused by overlapping tissues make it difficult to understand the internal conditions of the tissues in the obscured space.

[0003] At present, in order to enable doctors to accurately perceive the internal morphology of organs and tissues during laparoscopic surgery, the three-dimensional organ model constructed before the operation is usually holographically projected onto the three-dimensional organ model established in real time during the operation after alignment. This allows doctors to intuitively grasp the tissue situation in the organ-occluded space during the operation, achieve a high sense of presence during the operation, and enhance the doctor's perception of the organ and tissue during the operation.

[0004] In the existing technology, when aligning the three-dimensional organ model constructed before surgery and the three-dimensional organ model established in real time during surgery, a machine learning model is usually adopted to align the organ tissue model as a whole. The machine learning model needs to process three-dimensional image alignment information at the same alignment time. The amount of information processing is large, and the difficulty of feature processing increases, which naturally increases the structural complexity and computational complexity of the alignment model, resulting in a long alignment time, which makes it difficult to meet the real-time requirements for intraoperative alignment of organ models during surgery. Summary of the Invention

[0005] The purpose of the present invention is to provide a holographic fusion method and system for preoperative and intraoperative organ models based on three-dimensional registration, so as to solve the technical problem that in the existing technology, the registration of organ tissue models as a whole is performed, which increases the structural complexity and computational complexity of the registration model, resulting in a long registration time and difficulty in meeting the real-time requirements of intraoperative registration of organ models during surgery.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0007] A holographic fusion method for organ models before and during surgery based on three-dimensional registration, comprising the following steps:

[0008] Converting the preoperative organ model to be registered and the intraoperative organ model to be registered from three-dimensional point clouds into two-dimensional projections, respectively obtaining the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model, and the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model;

[0009] A GAN network is used to construct a 3D registration model for predicting the registration deformation field of the XY plane 2D projection, YZ plane 2D projection, and XZ plane 2D projection of the preoperative organ model;

[0010] Based on the registration deformation field output by the three-dimensional registration model, spatially transform the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model to obtain the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map of the preoperative organ model;

[0011] Performing three-dimensional reconstruction on the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, and the XZ plane two-dimensional projection registration map of the preoperative organ model to obtain a registration model of the preoperative organ model;

[0012] The registration model of the preoperative organ model is holographically projected onto the preoperative organ model to obtain the holographic fusion result of the preoperative and intraoperative organ models.

[0013] As a preferred solution of the present invention, the method for constructing the three-dimensional registration model includes:

[0014] The GAN network is used to construct an XY plane registration model, a YZ plane registration model, and an XZ plane registration model for registering the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model and the intraoperative organ model respectively;

[0015] A collaborative optimization module for controlling collaborative training of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model is set among the XY plane registration model, the YZ plane registration model, and the XZ plane registration model;

[0016] The collaborative optimization module is combined with the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to form a three-dimensional registration model for outputting the XY plane deformation field, the YZ plane deformation field, and the XZ plane deformation field of the preoperative organ model.

[0017] As a preferred embodiment of the present invention, the method for constructing the XY plane registration model includes:

[0018] The generator in the GAN network is used as the registration network, and the two-dimensional projection of the XY plane in the preoperative organ model and the two-dimensional projection of the XY plane in the intraoperative organ model are used as input, and the XY plane deformation field of the registered preoperative organ model is used as output;

[0019] The discriminator in the GAN network is a discriminant network, and the XY plane two-dimensional projection registration map obtained by spatially transforming the XY plane two-dimensional projection of the preoperative organ model by registering the XY plane deformation field of the preoperative organ model, and the XY plane two-dimensional projection of the intraoperative organ model are used as inputs, and the distinction result between the XY plane two-dimensional projection registration map and the XY plane two-dimensional projection of the intraoperative organ model is used as output;

[0020] The registration network is: Φ XY =G(M XY ,F XY ), where Φ XY To register the XY plane deformation field of the preoperative organ model, M XY is the two-dimensional projection of the XY plane in the preoperative organ model, F XY It is the two-dimensional projection of the XY plane in the intraoperative organ model;

[0021] The discriminant network is: F XY =D(M XY (Φ XY ),F XY ), where M XY (Φ XY ) is the XY plane two-dimensional projection registration map, F XY is the two-dimensional projection of the XY plane in the intraoperative organ model, P XY To M XY (Φ XY ) is judged as F XY probability;

[0022] The registration network Φ XY =G(M XY ,F XY ) and the discriminant network P XY =D(M XY (Φ XY ),F XY ) constitute the XY plane registration model;

[0023] The loss function of the XY plane registration model is: Where, L XY is the total loss of the XY plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,XY (Φ i,XY ) is the XY plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the XY plane registration model, Φ i,XY is the XY plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, Pi,XY For the discriminant network, M i,XY (Φ i,XY ) is judged as F i,XY The probability of F i,XY is the two-dimensional projection of the intraoperative organ model in the i-th training sample in the XY plane, M i,XY is the two-dimensional projection of the preoperative organ model in the XY plane in the i-th training sample, and n is the total number of training samples.

[0024] As a preferred solution of the present invention, the method for constructing the YZ plane registration model includes:

[0025] The generator in the GAN network is used as the registration network, and the two-dimensional YZ plane projection of the preoperative organ model and the two-dimensional YZ plane projection of the intraoperative organ model are used as input, and the YZ plane deformation field of the registered preoperative organ model is used as output;

[0026] The discriminator in the GAN network is a discriminant network, and the YZ plane two-dimensional projection registration map obtained by spatially transforming the YZ plane two-dimensional projection of the preoperative organ model by registering the YZ plane deformation field of the preoperative organ model, and the YZ plane two-dimensional projection of the intraoperative organ model are used as inputs, and the distinction result between the YZ plane two-dimensional projection registration map and the YZ plane two-dimensional projection of the intraoperative organ model is used as output;

[0027] The registration network is: Φ YZ =G(M YZ ,F YZ ), where Φ YZ To register the YZ plane deformation field of the preoperative organ model, M YZ is the two-dimensional projection of the YZ plane in the preoperative organ model, F YZ It is the two-dimensional projection of the YZ plane in the intraoperative organ model;

[0028] The discriminant network is: P YZ =D(M YZ (Φ YZ ),F YZ ), where M YZ (Φ YZ ) is the YZ plane two-dimensional projection registration map, F YZ is the two-dimensional projection of the YZ plane in the intraoperative organ model, P YZ To M YZ (Φ YZ ) is judged as F YZ probability;

[0029] The registration network Φ YZ =G(M YZ ,F YZ ) and the discriminant network P YZ=D(M YZ (Φ YZ ),F YZ ) constitute the YZ plane registration model;

[0030] The loss function of the YZ plane registration model is: Where, L YZ is the total loss of the YZ plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,YZ (Φ i,YZ ) is the YZ plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the YZ plane registration model, Φ i,YZ is the YZ plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,YZ For the discriminant network, M i,YZ (Φ i,YZ ) is judged as F i,YZ The probability of F i,YZ is the two-dimensional projection of the YZ plane in the intraoperative organ model in the i-th training sample, M i,YZ is the two-dimensional projection of the YZ plane in the preoperative organ model in the i-th training sample, and n is the total number of training samples.

[0031] As a preferred solution of the present invention, the method for constructing the XZ plane registration model includes:

[0032] The generator in the GAN network is used as the registration network, and the two-dimensional projection of the XZ plane in the preoperative organ model and the two-dimensional projection of the XZ plane in the intraoperative organ model are used as input, and the XZ plane deformation field of the registered preoperative organ model is used as output;

[0033] The discriminator in the GAN network is a discriminant network, and the XZ plane two-dimensional projection registration map obtained by spatially transforming the XZ plane two-dimensional projection of the preoperative organ model by registering the XZ plane deformation field of the preoperative organ model, and the XZ plane two-dimensional projection of the intraoperative organ model are used as inputs, and the distinction result between the XZ plane two-dimensional projection registration map and the XZ plane two-dimensional projection of the intraoperative organ model is used as output;

[0034] The registration network is: Φ XZ =G(M XZ ,F XZ ), where Φ XZ To register the XZ plane deformation field of the preoperative organ model, M XZ is the two-dimensional projection of the XZ plane in the preoperative organ model, F XZ It is the two-dimensional projection of the XZ plane in the intraoperative organ model;

[0035] The discriminant network is: P XZ =D(M XZ (Φ XZ ),F XZ ), where M XZ (Φ XZ ) is the XZ plane two-dimensional projection registration map, P XZ is the two-dimensional projection of the XZ plane in the intraoperative organ model, P XZ To M XZ (Φ XZ ) is judged as F XZ probability;

[0036] The registration network Φ XZ =G(M XZ ,F XZ ) and the discriminant network P XZ =D(M XZ (Φ XZ ),F XZ ) constitute the XZ plane registration model;

[0037] The loss function of the XZ plane registration model is: Where, L XZ is the total loss of the XZ plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,XZ (Φ i,XZ ) is the XZ plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the XZ plane registration model, Φ i,XZ is the XZ plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,XZ For the discriminant network, M i,XZ (Φ i,XZ ) is judged as F i,XZ The probability of F i,XZ is the two-dimensional projection of the XZ plane in the intraoperative organ model in the i-th training sample, M i,XZ is the two-dimensional projection of the preoperative organ model in the XZ plane in the i-th training sample, and n is the total number of training samples.

[0038] As a preferred solution of the present invention, the collaborative optimization module is: D similarity =min[H KL (L XY ||L YZ )+H KL (L YZ ||L XZ)+H KL (L XY ||L XZ )], where D similarity is the collaborative training optimization objective, min is the minimization operator, H KL (L XY ||L YZ ) is L XY and L YZ The KL divergence between KL (L YZ ||L XZ ) is L YZ and L XZ The KL divergence between KL (L XY ||L XZ ) is L XY and L XZ The KL divergence between

[0039] in,

[0040]

[0041] Where, L XZ is the total loss of the XZ plane registration model, L YZ is the total loss of the YZ plane registration model, L XY is the total loss of the XY plane registration model.

[0042] As a preferred solution of the present invention, the two-dimensional projection of the XY plane, the two-dimensional projection of the YZ plane, and the two-dimensional projection of the XZ plane are all obtained by orthogonal projection.

[0043] As a preferred solution of the present invention, the registration deformation field includes the XY plane deformation field, the YZ plane deformation field and the XZ plane deformation field.

[0044] As a preferred embodiment of the present invention, the present invention provides a preoperative and intraoperative organ model holographic fusion system based on three-dimensional registration, which is applied to a preoperative and intraoperative organ model holographic fusion method based on three-dimensional registration. The system includes:

[0045] a two-dimensional projection unit, configured to convert the preoperative organ model to be registered and the intraoperative organ model to be registered from three-dimensional point clouds into two-dimensional projections, and obtain the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model, as well as the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model;

[0046] A model construction unit, configured to construct a three-dimensional registration model for predicting the registration deformation field of the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model using a GAN network;

[0047] a projection registration unit, configured to perform spatial transformation on the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model based on the registration deformation field output by the three-dimensional registration model, to obtain an XY plane two-dimensional projection registration map, a YZ plane two-dimensional projection registration map, and an XZ plane two-dimensional projection registration map of the preoperative organ model;

[0048] A registration and reconstruction unit is used to perform three-dimensional reconstruction on the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, and the XZ plane two-dimensional projection registration map of the preoperative organ model to obtain a registration model of the preoperative organ model;

[0049] The holographic projection unit is used to holographically project the registration model of the preoperative organ model onto the preoperative organ model to obtain the holographic fusion result of the preoperative and intraoperative organ models.

[0050] As a preferred solution of the present invention, the three-dimensional registration model includes a collaborative optimization module, an XY plane registration model, a YZ plane registration model, and an XZ plane registration model, wherein:

[0051] The XY plane registration model, YZ plane registration model, and XZ plane registration model are all constructed using the GAN network;

[0052] The collaborative optimization module is used to control the collaborative training of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention refines the preoperative and intraoperative organ model registration from three-dimensional overall registration to multiple two-dimensional image registrations, and performs parallel registration processing on the two-dimensional images through three registration models, reducing the registration complexity and information processing volume, improving the registration efficiency, and meeting timeliness requirements. The collaborative optimization module maintains the synchronization of the three registration models in the two-dimensional image registration, thereby avoiding the reduction of registration accuracy due to deviations in the three registration models. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0056] Figure 1 A flow chart of a method for holographic fusion of organ models before and during surgery based on three-dimensional registration provided by an embodiment of the present invention;

[0057] Figure 2 A block diagram of a preoperative and intraoperative organ model holographic fusion system based on three-dimensional registration provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a three-dimensional registration model structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] like Figure 1 As shown, the present invention provides a holographic fusion method of organ models before and during surgery based on three-dimensional registration, comprising the following steps:

[0061] Converting the preoperative organ model to be registered and the intraoperative organ model to be registered from three-dimensional point clouds into two-dimensional projections, respectively obtaining the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model, as well as the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model;

[0062] A GAN network is used to construct a 3D registration model for predicting the registration deformation field of the XY plane 2D projection, YZ plane 2D projection, and XZ plane 2D projection of the preoperative organ model;

[0063] Based on the registration deformation field output by the three-dimensional registration model, the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model are spatially transformed to obtain the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map of the preoperative organ model;

[0064] Performing three-dimensional reconstruction on the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, and the XZ plane two-dimensional projection registration map of the preoperative organ model to obtain a registration model of the preoperative organ model;

[0065] The registration model of the preoperative organ model is holographically projected onto the preoperative organ model to obtain the holographic fusion result of the preoperative and intraoperative organ models.

[0066] The present invention reduces the registration of preoperative organ models and intraoperative organ models from a three-dimensional level to a two-dimensional level, and projects the three-dimensional model onto three planes through orthogonal projection to obtain an XY plane two-dimensional projection, a YZ plane two-dimensional projection, and an XZ plane two-dimensional projection, thereby dividing the registration processing process of the traditional stereo registration model for the three-dimensional organ model into an image registration processing process of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model. The registration data is reduced from three-dimensional data to two-dimensional data, thereby reducing the amount of registration processing data. The registration efficiency of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model is improved compared to the traditional stereo registration model.

[0067] The image registration processing processes of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model are implemented in parallel. Therefore, the three-dimensional registration model constructed by the present invention can reduce the dimensionality of the three-dimensional model registration into three parallel two-dimensional image registrations, thereby improving the registration efficiency. Moreover, it is divided into three parallel two-dimensional image registrations. Each plane registration model can focus on the registration of the corresponding two-dimensional image. Compared with the traditional stereo registration model that needs to take into account the three-dimensional data level, the plane registration model does not need to take into account the three-dimensional data, thereby improving the fine granularity of the registration, thereby ensuring the fineness of the registration, and ultimately improving the accuracy of the registration.

[0068] The present invention adopts a GAN network to construct the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to ensure the consistency of the registration plane structure, thereby providing a structural basis for the parallelism of the model. The generator in the GAN network is used as the registration network to generate a registration deformation field, so as to obtain the registered image through the spatial transformation of the registration deformation field. The discriminator in the GAN network is used as a discriminant network to evaluate the registration performance of the registration network. Through adversarial training of the registration network and the discriminant network, the discriminant network urges the registration network to generate the registered image (XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, XZ plane two-dimensional projection registration map) and match it with the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model, that is, the registration accuracy is optimized.

[0069] The present invention sets the loss functions of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model into two parts. The first part is the registration loss of the registration network, and the second part is the adversarial loss between the registration network and the discriminant network. The registration loss of the registration network quantifies the degree of correspondence difference between the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, the XZ plane two-dimensional projection registration map and the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The lower the degree of difference, the higher the registration accuracy.

[0070] The adversarial loss of the registration network and the discriminant network: For the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map generated by the registration network, the discrimination value of the discriminant network is close to 1, that is, the discriminant network is misled to misjudge the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map as the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The adversarial loss term XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map are the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The difference between the two-dimensional projection of the surface and the two-dimensional projection of the XZ plane is used to urge the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map generated by the registration network to match the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map generated by the combination of the two are highly similar to the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The discriminant network cannot accurately distinguish them, and the output of the registration network is the expected optimal deformation field.

[0071] In summary, the XY plane registration model, YZ plane registration model, and XZ plane registration model constructed by the present invention can achieve high-precision registration on the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection, so that the registration model of the preoperative organ model obtained after reconstruction inherits the high-precision advantage.

[0072] In order to avoid the deviation of registration in the two-dimensional layer splitting of three dimensions, the present invention sets a collaborative optimization module between the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to maintain the mutual coordination between the XY plane registration model, the YZ plane registration model, and the XZ plane registration model, so as to avoid the registration deviation, thereby avoiding the reduction in the accuracy of the registration model reconstructed as the preoperative organ model. The collaborative optimization target in the collaborative optimization module is to minimize the difference between the total loss of the XZ plane registration model, the total loss of the YZ plane registration model, and the total loss of the XY plane registration model, which can ensure that the XY plane registration model, the YZ plane registration model, and the XZ plane registration model synchronously reach a high-precision training effect, thereby realizing the XY plane registration model, the YZ plane registration model, and the XZ plane registration model synchronously outputting high-precision registration results.

[0073] While the parallel registration operations of the XY plane registration model, YZ plane registration model, and XZ plane registration model improve the registration efficiency, it can also ensure the synchronization of the two-dimensional reconstruction into the three-dimensional form after registration, avoid the asynchronous deviation between the models caused by the three-dimensional division into two-dimensional registration, and ensure the feasibility of reducing the dimensionality of three-dimensional stereo registration to two-dimensional image registration.

[0074] like Figure 3 As shown, the method for constructing a three-dimensional registration model includes:

[0075] The GAN network is used to construct an XY plane registration model, a YZ plane registration model, and an XZ plane registration model for registering the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model and the intraoperative organ model respectively;

[0076] A collaborative optimization module for controlling collaborative training of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model is set among the XY plane registration model, the YZ plane registration model, and the XZ plane registration model;

[0077] The collaborative optimization module is combined with the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to form a three-dimensional registration model for outputting the XY plane deformation field, the YZ plane deformation field, and the XZ plane deformation field of the preoperative organ model.

[0078] The present invention adopts a GAN network to construct the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to ensure the consistency of the registration plane structure, thereby providing a structural basis for the parallelism of the model. The generator in the GAN network is used as the registration network to generate a registration deformation field, so as to obtain a registered image through the spatial transformation of the registration deformation field. The discriminator in the GAN network is used as a discriminant network to evaluate the registration performance of the registration network. Through adversarial training between the registration network and the discriminant network, the discriminant network urges the registration network to generate a registered image (XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, XZ plane two-dimensional projection registration map) that matches the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model, that is, the registration accuracy is optimized, as follows:

[0079] The construction method of the XY plane registration model includes:

[0080] The generator in the GAN network is used as the registration network, and the two-dimensional projection of the XY plane in the preoperative organ model and the two-dimensional projection of the XY plane in the intraoperative organ model are used as input, and the XY plane deformation field of the registered preoperative organ model is used as output;

[0081] The discriminator in the GAN network is used as the discriminant network, and the XY plane 2D projection registration map obtained by spatially transforming the XY plane 2D projection of the preoperative organ model by registering the XY plane deformation field of the preoperative organ model and the XY plane 2D projection of the intraoperative organ model are used as inputs, and the distinction result between the XY plane 2D projection registration map and the XY plane 2D projection of the intraoperative organ model is used as output;

[0082] The registration network is: Φ XY =G(M XY ,F XY ), where Φ XY To register the XY plane deformation field of the preoperative organ model, M XY is the two-dimensional projection of the XY plane in the preoperative organ model, F XY It is the two-dimensional projection of the XY plane in the intraoperative organ model;

[0083] The discriminant network is: P XY =D(M XY (Φ XY ),F XY ), where M XY (Φ XY ) is the XY plane two-dimensional projection registration map, F XY is the two-dimensional projection of the XY plane in the intraoperative organ model, P XY To M XY (Φ XY ) is judged as F XY probability;

[0084] Registration network Φ XY =G(M XY ,F XY ) and the discriminant network P XY =D(M XY (Φ XY ),F XY ) constitutes an XY plane registration model;

[0085] The loss function of the XY plane registration model is: Where, L XY is the total loss of the XY plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,XY (Φ i,XY ) is the XY plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the XY plane registration model, Φ i,XY is the XY plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,XY For the discriminant network, Mi,XY (Φ i,XY ) is judged as F i,XY The probability of F i,XY is the two-dimensional projection of the intraoperative organ model in the i-th training sample in the XY plane, M i,XY is the two-dimensional projection of the preoperative organ model in the XY plane in the i-th training sample, and n is the total number of training samples.

[0086] The construction method of the YZ plane registration model includes:

[0087] The generator in the GAN network is used as the registration network, and the two-dimensional YZ plane projection of the preoperative organ model and the two-dimensional YZ plane projection of the intraoperative organ model are used as input, and the YZ plane deformation field of the registered preoperative organ model is used as output;

[0088] The discriminator in the GAN network is used as the discriminant network, and the YZ plane 2D projection registration map obtained by spatially transforming the YZ plane 2D projection of the preoperative organ model by registering the YZ plane deformation field of the preoperative organ model, and the YZ plane 2D projection of the intraoperative organ model are used as inputs, and the distinction result between the YZ plane 2D projection registration map and the YZ plane 2D projection of the intraoperative organ model is used as output;

[0089] The registration network is: Φ YZ =G(M YZ ,F YZ ), where Φ YZ To register the YZ plane deformation field of the preoperative organ model, M YZ is the two-dimensional projection of the YZ plane in the preoperative organ model, F YZ It is the two-dimensional projection of the YZ plane in the intraoperative organ model;

[0090] The discriminant network is: P YZ =D(M YZ (Φ YZ ),F YZ ), where M YZ (Φ YZ ) is the YZ plane two-dimensional projection registration map, F YZ is the two-dimensional projection of the YZ plane in the intraoperative organ model, P YZ To M YZ (Φ YZ ) is judged as F YZ probability;

[0091] Registration network Φ YZ =G(M YZ ,F YZ ) and the discriminant network P YZ =D(M YZ (Φ YZ ),F YZ) constitutes a YZ plane registration model;

[0092] The loss function of the YZ plane registration model is: Where, L YZ is the total loss of the YZ plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,YZ (Φ i,YZ ) is the YZ plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the YZ plane registration model, Φ i,YZ is the YZ plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,YZ For the discriminant network, M i,YZ (Φ i,YZ ) is judged as F i,YZ The probability of F i,YZ is the two-dimensional projection of the YZ plane in the intraoperative organ model in the i-th training sample, M i,YZ is the two-dimensional projection of the YZ plane in the preoperative organ model of the i-th training sample, and n is the total number of training samples.

[0093] The construction method of the XZ plane registration model includes:

[0094] The generator in the GAN network is used as the registration network, and the two-dimensional projection of the XZ plane in the preoperative organ model and the two-dimensional projection of the XZ plane in the intraoperative organ model are used as input, and the XZ plane deformation field of the registered preoperative organ model is used as output;

[0095] The discriminator in the GAN network is used as the discriminant network, and the XZ plane 2D projection registration map obtained by spatially transforming the XZ plane 2D projection of the preoperative organ model by registering the XZ plane deformation field of the preoperative organ model, and the XZ plane 2D projection of the intraoperative organ model are used as inputs, and the distinction result between the XZ plane 2D projection registration map and the XZ plane 2D projection of the intraoperative organ model is used as output;

[0096] The registration network is: Φ XZ =G(M XZ ,F XZ ), where Φ XZ To register the XZ plane deformation field of the preoperative organ model, M XZ is the two-dimensional projection of the XZ plane in the preoperative organ model, F XZ It is the two-dimensional projection of the XZ plane in the intraoperative organ model;

[0097] The discriminant network is: P XZ =D(M XZ (Φ XZ ),FXZ ), where M XZ (Φ XZ ) is the XZ plane two-dimensional projection registration map, F XZ is the two-dimensional projection of the XZ plane in the intraoperative organ model, P XZ To M XZ (Φ XZ ) is judged as F XZ probability;

[0098] Registration network Φ XZ =G(M XZ ,F XZ ) and the discriminant network P XZ =D(M XZ (Φ XZ ),F XZ ) constitutes an XZ plane registration model;

[0099] The loss function of the XZ plane registration model is: Where, L XZ is the total loss of the XZ plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,XZ (Φ i,XZ ) is the XZ plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the XZ plane registration model, Φ i,XZ is the XZ plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,XZ For the discriminant network, M i,XZ (Φ i,XZ ) is judged as F i,XZ The probability of F i,XZ is the two-dimensional projection of the XZ plane in the intraoperative organ model in the i-th training sample, M i,XZ is the two-dimensional projection of the XZ plane in the preoperative organ model in the i-th training sample, and n is the total number of training samples.

[0100] The present invention sets the loss functions of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model into two parts. The first part is the registration loss of the registration network, and the second part is the adversarial loss between the registration network and the discriminant network. The registration loss of the registration network quantifies the degree of correspondence difference between the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, the XZ plane two-dimensional projection registration map and the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The lower the degree of difference, the higher the registration accuracy.

[0101] The adversarial loss of the registration network and the discriminant network: For the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map generated by the registration network, the discrimination value of the discriminant network is close to 1, that is, the discriminant network is misled to misjudge the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map as the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The adversarial loss term XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map are the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The difference between the two-dimensional projection of the surface and the two-dimensional projection of the XZ plane is used to urge the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map generated by the registration network to match the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map generated by the combination of the two are highly similar to the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model. The discriminant network cannot accurately distinguish them, and the output of the registration network is the expected optimal deformation field.

[0102] The XY plane registration model, YZ plane registration model, and XZ plane registration model constructed by the present invention can achieve high-precision registration on the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection, so that the registration model of the preoperative organ model obtained after reconstruction inherits the high-precision advantage.

[0103] In order to avoid the deviation of registration in the two-dimensional layer splitting of three dimensions, the present invention sets a collaborative optimization module between the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to maintain the mutual coordination between the XY plane registration model, the YZ plane registration model, and the XZ plane registration model, so that there will be no registration deviation, thereby avoiding the reduction in the accuracy of the registration model reconstructed as the preoperative organ model. The collaborative optimization target in the collaborative optimization module is to minimize the difference between the total loss of the XZ plane registration model, the total loss of the YZ plane registration model, and the total loss of the XY plane registration model, so as to ensure that the XY plane registration model, the YZ plane registration model, and the XZ plane registration model synchronously reach a high-precision training effect, thereby realizing the XY plane registration model, the YZ plane registration model, and the XZ plane registration model synchronously outputting high-precision registration results, as follows:

[0104] The collaborative optimization module is: D similarity =min[H KL (L XY ||L YZ )+H KL (L YZ||L XZ )+H KL (L XY ||L XZ )], where D similarity is the collaborative training optimization objective, min is the minimization operator, H KL (L XY ||L YZ ) is L XY and L YZ The KL divergence between KL (L YZ ||L XZ ) is L YZ and L XZ The KL divergence between KL (L XY ||L XZ ) is L XY and L XZ The KL divergence between

[0105] in,

[0106]

[0107] Where, L XZ is the total loss of the XZ plane registration model, L YZ is the total loss of the YZ plane registration model, L XY is the total loss of the XY plane registration model.

[0108] The two-dimensional projection of the XY plane, the two-dimensional projection of the YZ plane, and the two-dimensional projection of the XZ plane are all obtained by orthogonal projection.

[0109] The registration deformation field includes the XY plane deformation field, the YZ plane deformation field and the XZ plane deformation field.

[0110] Collect preoperative organ tissue images and perform 3D reconstruction based on the preoperative organ tissue images using 3D Slicer to obtain a preoperative organ model;

[0111] The intraoperative organ tissue is collected by a binocular camera to obtain a binocular image of the intraoperative organ, and a semi-supervised binocular image depth estimation model is used to perform binocular image depth estimation based on the binocular image of the intraoperative organ to obtain an intraoperative organ depth map;

[0112] The intraoperative organ depth map is used to generate an intraoperative organ point cloud, and a three-dimensional reconstruction is performed based on the intraoperative organ point cloud to obtain an intraoperative organ model.

[0113] The present invention uses a semi-supervised binocular image depth estimation model to perform stereo depth estimation on binocular images of organs during surgery. The semi-supervised binocular image depth estimation model includes a two-branch convolutional neural network and bidirectional adaptive supervision to ensure that accurate left and right disparity can be obtained based on the binocular images. The details are as follows:

[0114] The method for constructing a semi-supervised binocular image depth estimation model includes:

[0115] The left and right images in the intraoperative organ binocular image are used to estimate the bidirectional disparity using a two-branch convolutional neural network to obtain the left and right bidirectional disparity;

[0116] A reconstruction loss and consistency loss for left-right bidirectional disparity are established, and a two-branch convolutional neural network is trained in a bidirectional self-supervised manner based on the reconstruction loss and consistency loss for left-right bidirectional disparity to obtain a semi-supervised binocular image depth estimation model.

[0117] The semi-supervised binocular image depth estimation model is:

[0118] d l =CNN1(I l );

[0119] d r =CNN2(I r );

[0120] Where, d l is the left disparity, d r is the right parallax, I l is the left eye image, I r is the right eye image, CNN1 is the first branch convolutional neural network, and CNN2 is the second branch convolutional neural network;

[0121] The reconstruction loss is:

[0122]

[0123] Where, I res is the reconstruction loss, d l,i is the left disparity of the i-th binocular image in the dataset used to train the semi-supervised binocular image depth estimation model, I l,i is the left eye image of the i-th binocular image in the dataset, I r,i is the right eye image of the i-th binocular image in the dataset, d r,i is the right disparity of the i-th binocular image in the dataset, warping(d l,i ,I l,i ) is 1 l,i By d l,i The reconstructed right image, warping(d r,i,I r,i ) is 1 r,i By d r,i The reconstructed left eye image, n is the total number of binocular images in the dataset;

[0124] The dual-branch convolutional neural network includes a first-branch convolutional neural network and a second-branch convolutional neural network. The first-branch convolutional neural network is used to predict the left disparity of the left-eye image relative to the right-eye image, and the second-branch convolutional neural network is used to predict the right disparity of the right-eye image relative to the left-eye image.

[0125] Bidirectional adaptive supervision includes bidirectional supervision of reconstruction loss and bidirectional supervision of consistency loss. In the bidirectional supervision of reconstruction loss, one direction of supervision is the loss between the original left-eye image and the right-eye image obtained by reconstructing the left disparity of the left-eye image relative to the right-eye image (reconstruction loss of left disparity). The other direction of supervision is the loss between the original right-eye image and the original left-eye image obtained by reconstructing the right disparity of the right-eye image relative to the left-eye image (reconstruction loss of right disparity).

[0126] The bidirectional supervision of the reconstruction loss is quantified by the reconstruction loss between the original left image and the original right image. The original left image and the original right image are the label values ​​of the reconstruction loss of the two-branch convolutional neural network, and the label value is the true value. Therefore, the bidirectional supervision of the reconstruction loss belongs to supervised training with accurate supervision truth value for the two-branch convolutional neural network.

[0127] The consistency loss is:

[0128]

[0129] Where, d res is the consistency loss, warping(d l,i ,d l,i ) is d l,i By d l,i The reconstructed right disparity, warping(d r,i ,d r,i ) is d r,i By d r,i The reconstructed left disparity, n is the total number of binocular images in the dataset, d r,i The parallax gradient, d l,i parallax gradient.

[0130] One direction of supervision in the two-way supervision of disparity consistency loss is the loss between the left disparity predicted by the first branch neural network, which is reconstructed by the left disparity predicted by the first branch neural network, and the right disparity predicted by the second branch neural network (left-right disparity consistency loss). The other direction of supervision is the loss between the left disparity predicted by the second branch neural network, which is reconstructed by the right disparity predicted by the second branch neural network, and the left disparity predicted by the first branch neural network (right-left disparity consistency loss).

[0131] The bidirectional supervision of disparity consistency loss is quantified by the reconstruction loss between the left disparity predicted by the first branch neural network and the right disparity predicted by the second branch neural network. The left disparity and the right disparity are the label values ​​of the consistency loss of the two-branch convolutional neural network, but the label value is a predicted value. Therefore, the bidirectional supervision of the reconstruction loss belongs to unsupervised training of the two-branch convolutional neural network without supervised truth value.

[0132] The present invention also superimposes the disparity gradient in the disparity consistency loss, thereby ensuring the left and right disparity consistency with the highest disparity value obtained after training is completed, while also having the smallest disparity gradient, so that the obtained disparity map has the highest density and the disparity is kept smooth locally.

[0133] The combination of reconstruction loss and consistency loss enables the combination of supervised and unsupervised training, resulting in a semi-supervised two-branch convolutional neural network. This constructs a semi-supervised binocular image depth estimation model, which can obtain accurate left and right disparity estimates. The disparity can be directly converted into depth, thus enabling accurate stereo depth estimation.

[0134] After obtaining the left and right bidirectional parallax, the present invention converts it into the corresponding two organ point cloud data. By point cloud registration and optimizing the construction of the intraoperative organ surface point cloud, an accurate intraoperative organ three-dimensional model is obtained, as follows:

[0135] The methods for constructing intraoperative organ depth maps include:

[0136] Input the binocular image of the intraoperative organ into the semi-supervised binocular image depth estimation model to obtain the left and right bidirectional disparity of the binocular image of the intraoperative organ;

[0137] Obtaining the baseline distance and focal length of the binocular camera, and converting the left parallax in the left and right bidirectional parallax to obtain a first intraoperative organ depth map;

[0138] The first intraoperative organ depth map is: h1 = bf / d l ;

[0139] Where h1 is the first intraoperative organ depth map, b is the baseline distance, f is the focal length, and d is the l is the left parallax;

[0140] Obtain the baseline distance and focal length of the binocular camera, and obtain the second intraoperative organ depth map based on the right parallax in the left and right bidirectional parallax;

[0141] The second intraoperative organ depth map is: h2 = bf / d r ;

[0142] Where h2 is the second intraoperative organ depth map, b is the baseline distance, f is the focal length, and d is the r is the right parallax.

[0143] Methods for generating intraoperative organ point clouds include:

[0144] Get the camera intrinsic parameters of the binocular camera;

[0145] The camera intrinsic parameters and the first intraoperative organ depth map are used to perform point cloud conversion to obtain the first intraoperative organ point cloud (X1, Y1, Z1), where X1 is the X-axis coordinate in the three-dimensional coordinates of the point cloud, Y1 is the Y-axis coordinate in the three-dimensional coordinates of the point cloud, and Z1 is the Z-axis coordinate in the three-dimensional coordinates of the point cloud;

[0146] in, Z1=D(u1,v1);

[0147] Where (u1, v1) is the pixel coordinate in the depth map h1, f x , f y are camera principal distances, c x , c y are the coordinates of the camera principal point, and D(u1,v1) is the depth value at (u1,v1) in the depth map h1;

[0148] The camera internal parameters and the depth map of the second intraoperative organ are used to perform point cloud conversion to obtain the second intraoperative organ point cloud (X2, Y2, Z2), where X2 is the X-axis coordinate in the three-dimensional coordinates of the point cloud, Y2 is the Y-axis coordinate in the three-dimensional coordinates of the point cloud, and Z2 is the Z-axis coordinate in the three-dimensional coordinates of the point cloud;

[0149] in, Z2=D(u1,v1);

[0150] Where (u2, v2) is the pixel coordinate in the depth map h2, f x , f y are camera principal distances, c x , c y are the coordinates of the camera principal point, and D(u2,v2) is the depth at (u2,v2) in the depth map h2.

[0151] Methods for intraoperative organ model reconstruction include:

[0152] The first intraoperative organ point cloud and the second intraoperative organ point cloud are aligned and registered using the ICP point cloud registration method to obtain an intraoperative organ registration point cloud;

[0153] The intraoperative organ registration point cloud is reconstructed into three dimensions using the Poisson reconstruction method to obtain the intraoperative organ model;

[0154] The image optimization tool in SLAM technology is used to optimize the intraoperative organ model.

[0155] like Figure 2 As shown, the present invention provides a preoperative and intraoperative organ model holographic fusion system based on three-dimensional registration, which is applied to a preoperative and intraoperative organ model holographic fusion method based on three-dimensional registration. The system includes:

[0156] a two-dimensional projection unit, configured to convert the preoperative organ model and the intraoperative organ model to be registered from three-dimensional point clouds into two-dimensional projections, and obtain the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model, as well as the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model;

[0157] A model construction unit, configured to construct a three-dimensional registration model for predicting the registration deformation field of the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model using a GAN network;

[0158] A projection registration unit is used to perform spatial transformation on the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model based on the registration deformation field output by the three-dimensional registration model, so as to obtain the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map of the preoperative organ model;

[0159] A registration and reconstruction unit is used to perform three-dimensional reconstruction on the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, and the XZ plane two-dimensional projection registration map of the preoperative organ model to obtain a registration model of the preoperative organ model;

[0160] The holographic projection unit is used to holographically project the registration model of the preoperative organ model onto the preoperative organ model to obtain the holographic fusion result of the preoperative and intraoperative organ models.

[0161] The 3D registration model includes collaborative optimization module, XY plane registration model, YZ plane registration model, and XZ plane registration model, among which,

[0162] The XY plane registration model, YZ plane registration model, and XZ plane registration model are all constructed using the GAN network;

[0163] The collaborative optimization module is used to control the collaborative training of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model.

[0164] The present invention refines the preoperative and intraoperative organ model registration from three-dimensional overall registration to multiple two-dimensional image registrations, and performs parallel registration processing on the two-dimensional images through three registration models, reducing the registration complexity and information processing volume, improving the registration efficiency, and meeting timeliness requirements. The collaborative optimization module maintains the synchronization of the three registration models in the two-dimensional image registration, thereby avoiding the reduction of registration accuracy due to deviations in the three registration models.

[0165] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A holographic fusion method for organ models before and during surgery based on three-dimensional registration, characterized in that: The following steps are involved: Converting the preoperative organ model to be registered and the intraoperative organ model to be registered from three-dimensional point clouds into two-dimensional projections, respectively obtaining the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model, and the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model; A GAN network is used to construct a 3D registration model for predicting the registration deformation field of the XY plane 2D projection, YZ plane 2D projection, and XZ plane 2D projection of the preoperative organ model. Based on the registration deformation field output by the three-dimensional registration model, spatially transform the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model to obtain the XY plane two-dimensional projection registration map, YZ plane two-dimensional projection registration map, and XZ plane two-dimensional projection registration map of the preoperative organ model; Performing three-dimensional reconstruction on the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, and the XZ plane two-dimensional projection registration map of the preoperative organ model to obtain a registration model of the preoperative organ model; The registration model of the preoperative organ model is holographically projected onto the preoperative organ model to obtain the holographic fusion result of the preoperative and intraoperative organ models.

2. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The method for constructing the three-dimensional registration model includes: The GAN network is used to construct an XY plane registration model, a YZ plane registration model, and an XZ plane registration model for registering the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model and the intraoperative organ model respectively; A collaborative optimization module for controlling collaborative training of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model is set among the XY plane registration model, the YZ plane registration model, and the XZ plane registration model; The collaborative optimization module is combined with the XY plane registration model, the YZ plane registration model, and the XZ plane registration model to form a three-dimensional registration model for outputting the XY plane deformation field, the YZ plane deformation field, and the XZ plane deformation field of the preoperative organ model.

3. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The method for constructing the XY plane registration model includes: The generator in the GAN network is used as the registration network, and the two-dimensional projection of the XY plane in the preoperative organ model and the two-dimensional projection of the XY plane in the intraoperative organ model are used as input, and the XY plane deformation field of the registered preoperative organ model is used as output; The discriminator in the GAN network is a discriminant network, and the XY plane two-dimensional projection registration map obtained by spatially transforming the XY plane two-dimensional projection of the preoperative organ model by registering the XY plane deformation field of the preoperative organ model, and the XY plane two-dimensional projection of the intraoperative organ model are used as inputs, and the distinction result between the XY plane two-dimensional projection registration map and the XY plane two-dimensional projection of the intraoperative organ model is used as output; The registration network is: Φ XY =G(M XY ,F XY ), where Φ XY To register the XY plane deformation field of the preoperative organ model, M XY is the two-dimensional projection of the XY plane in the preoperative organ model, F XY It is the two-dimensional projection of the XY plane in the intraoperative organ model; The discriminant network is: P XY =D(M XY (Φ XY ),F XY ), where M XY (Φ XY ) is the XY plane two-dimensional projection registration map, F XY is the two-dimensional projection of the XY plane in the intraoperative organ model, P XY To M XY (Φ XY ) is judged as F XY probability; The registration network Φ XY =G(M XY ,F XY ) and the discriminant network P XY =D(M XY (Φ XY ),F XY ) constitute the XY plane registration model; The loss function of the XY plane registration model is: Where, L XY is the total loss of the XY plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,XY (Φ i,XY ) is the XY plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the XY plane registration model, Φ i,XY is the XY plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,XY For the discriminant network, M i,XY (Φ i,XY ) is judged as F i,XY The probability of F i,XY is the two-dimensional projection of the intraoperative organ model in the i-th training sample in the XY plane, M i,XY is the two-dimensional projection of the preoperative organ model in the XY plane in the i-th training sample, and n is the total number of training samples.

4. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The method for constructing the YZ plane registration model includes: The generator in the GAN network is used as the registration network, and the two-dimensional YZ plane projection of the preoperative organ model and the two-dimensional YZ plane projection of the intraoperative organ model are used as input, and the YZ plane deformation field of the registered preoperative organ model is used as output; The discriminator in the GAN network is a discriminant network, and the YZ plane two-dimensional projection registration map obtained by spatially transforming the YZ plane two-dimensional projection of the preoperative organ model by registering the YZ plane deformation field of the preoperative organ model, and the YZ plane two-dimensional projection of the intraoperative organ model are used as inputs, and the distinction result between the YZ plane two-dimensional projection registration map and the YZ plane two-dimensional projection of the intraoperative organ model is used as output; The registration network is: Φ YZ =G(M YZ ,F YZ ), where Φ YZ To register the YZ plane deformation field of the preoperative organ model, M YZ is the two-dimensional projection of the YZ plane in the preoperative organ model, F YZ It is the two-dimensional projection of the YZ plane in the intraoperative organ model; The discriminant network is: P YZ =D(M YZ (Φ YZ ),F YZ ), where M YZ (Φ YZ ) is the YZ plane two-dimensional projection registration map, F YZ is the two-dimensional projection of the YZ plane in the intraoperative organ model, P YZ To M YZ (Φ YZ ) is judged as F YZ probability; The registration network Φ YZ =G(M YZ ,F YZ ) and the discriminant network P YZ =D(M YZ (Φ YZ ),F YZ ) constitute the YZ plane registration model; The loss function of the YZ plane registration model is: Where, L YZ is the total loss of the YZ plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,YZ (Φ i,YZ ) is the YZ plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the YZ plane registration model, Φ i,YZ is the YZ plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,YZ For the discriminant network, M i,YZ (Φ i,YZ ) is judged as F i,YZ The probability of F i,YZ is the two-dimensional projection of the YZ plane in the intraoperative organ model in the i-th training sample, M i,YZ is the two-dimensional projection of the YZ plane in the preoperative organ model in the i-th training sample, and n is the total number of training samples.

5. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The method for constructing the XZ plane registration model includes: The generator in the GAN network is used as the registration network, and the two-dimensional projection of the XZ plane in the preoperative organ model and the two-dimensional projection of the XZ plane in the intraoperative organ model are used as input, and the XZ plane deformation field of the registered preoperative organ model is used as output; The discriminator in the GAN network is a discriminant network, and the XZ plane two-dimensional projection registration map obtained by spatially transforming the XZ plane two-dimensional projection of the preoperative organ model by registering the XZ plane deformation field of the preoperative organ model, and the XZ plane two-dimensional projection of the intraoperative organ model are used as inputs, and the distinction result between the XZ plane two-dimensional projection registration map and the XZ plane two-dimensional projection of the intraoperative organ model is used as output; The registration network is: Φ XZ =G(M XZ ,F XZ ), where Φ XZ To register the XZ plane deformation field of the preoperative organ model, M XZ is the two-dimensional projection of the XZ plane in the preoperative organ model, F XZ It is the two-dimensional projection of the XZ plane in the intraoperative organ model; The discriminant network is: P XZ =D(M XZ (Φ XZ ),F XZ ), where M XZ (Φ XZ ) is the XZ plane two-dimensional projection registration map, F XZ is the two-dimensional projection of the XZ plane in the intraoperative organ model, P XZ To M XZ (Φ XZ ) is judged as F XZ probability; The registration network Φ XZ =G(M XZ ,F XZ ) and the discriminant network P XZ =D(M XZ (Φ XZ ),F XZ ) constitute the XZ plane registration model; The loss function of the XZ plane registration model is: Where, L XZ is the total loss of the XZ plane registration model, is the registration loss of the registration network, is the adversarial loss between the discriminative network and the registration network, M i,XZ (Φ i,XZ ) is the XZ plane two-dimensional projection registration map obtained by the registration network for the i-th training sample used to train the XZ plane registration model, Φ i,XZ is the XZ plane deformation field of the pre-registration organ model obtained by the registration network for the i-th training sample, P i,XZ For the discriminant network, M i,XZ (Φ i,XZ ) is judged as F i,XZ The probability of F i,XZ is the two-dimensional projection of the XZ plane in the intraoperative organ model in the i-th training sample, M i,XZ is the two-dimensional projection of the preoperative organ model in the XZ plane in the i-th training sample, and n is the total number of training samples.

6. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The collaborative optimization module is: similarity =min[H KL (L XY ||L YZ )+H KL (L YZ ||L XZ )+H KL (L XY ||L XZ )], where D similarity is the collaborative training optimization objective, min is the minimization operator, H KL (L XY ||L YZ ) is L XY and L YZ The KL divergence between KL (L YZ ||L XZ ) is L YZ and L XZ The KL divergence between KL (L XY ||L XZ ) is L XY and L XZ The KL divergence between in, Where, L XZ is the total loss of the XZ plane registration model, L YZ is the total loss of the YZ plane registration model, L XY is the total loss of the XY plane registration model.

7. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The two-dimensional projection of the XY plane, the two-dimensional projection of the YZ plane, and the two-dimensional projection of the XZ plane are all obtained by orthogonal projection.

8. The method for holographic fusion of organ models before and during surgery based on three-dimensional registration according to claim 1, characterized in that: The registration deformation field includes the XY plane deformation field, the YZ plane deformation field and the XZ plane deformation field.

9. A holographic fusion system for organ models before and during surgery based on three-dimensional registration, characterized by: The method for holographic fusion of preoperative and intraoperative organ models based on three-dimensional registration according to any one of claims 1 to 8, wherein the system comprises: a two-dimensional projection unit, configured to convert the preoperative organ model to be registered and the intraoperative organ model to be registered from three-dimensional point clouds into two-dimensional projections, and obtain the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the preoperative organ model, as well as the XY plane two-dimensional projection, YZ plane two-dimensional projection, and XZ plane two-dimensional projection of the intraoperative organ model; A model construction unit, configured to construct a three-dimensional registration model for predicting the registration deformation field of the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model using a GAN network; a projection registration unit, configured to perform spatial transformation on the XY plane two-dimensional projection, the YZ plane two-dimensional projection, and the XZ plane two-dimensional projection of the preoperative organ model based on the registration deformation field output by the three-dimensional registration model, to obtain an XY plane two-dimensional projection registration map, a YZ plane two-dimensional projection registration map, and an XZ plane two-dimensional projection registration map of the preoperative organ model; A registration and reconstruction unit is used to perform three-dimensional reconstruction on the XY plane two-dimensional projection registration map, the YZ plane two-dimensional projection registration map, and the XZ plane two-dimensional projection registration map of the preoperative organ model to obtain a registration model of the preoperative organ model; The holographic projection unit is used to holographically project the registration model of the preoperative organ model onto the preoperative organ model to obtain the holographic fusion result of the preoperative and intraoperative organ models.

10. The preoperative and intraoperative organ model holographic fusion system based on three-dimensional registration according to claim 9, characterized in that: The three-dimensional registration model includes a collaborative optimization module, an XY plane registration model, a YZ plane registration model, and an XZ plane registration model, wherein: The XY plane registration model, YZ plane registration model, and XZ plane registration model are all constructed using the GAN network; The collaborative optimization module is used to control the collaborative training of the XY plane registration model, the YZ plane registration model, and the XZ plane registration model.

Images