A soft tissue adaptive deformation method based on mixed reality and 3DGAN

Abstract

The present invention provides a method and system for adaptive deformation of soft tissue based on mixed reality and 3DGAN, belonging to the field of medical technology. The method of the present invention comprises the following steps: obtaining a global three-dimensional model before surgery and converting it into point cloud data; obtaining the depth map data of the local image of the patient during surgery in real time and converting it into point cloud data; sending the acquired point cloud data into a nested 3DGAN network model to realize the mapping of local deformation to global deformation; converting the obtained deformed point cloud global data into a three-dimensional model. The present invention also provides a system for realizing the above-mentioned method for adaptive deformation of soft tissue based on mixed reality and 3DGAN. The beneficial effects of the present invention are: during surgery, the deformation of human soft tissue during surgery can be matched with the virtual three-dimensional model before surgery according to the local image of the patient, and a three-dimensional model of the deformed soft tissue in the patient's body can be generated in real time.

Description

Technical Field

[0001] The present invention relates to a medical technology, and in particular to a soft tissue adaptive deformation method based on mixed reality and 3D GAN. Background Art

[0002] During traditional surgery, doctors need to combine professional knowledge with the patient's preoperative CT or MR scanned two-dimensional images to conceive a three-dimensional model of the patient's body during surgery, which is a great test of the doctor's experience and the energy consumption of long-term high-load surgery. The intraoperative navigation system is a relatively new surgical assistance method. It matches the patient's body three-dimensional model with the intraoperative anatomical structure through a registration method, and then overlaps the virtual three-dimensional model with the real patient and displays it to the doctor through the corresponding imaging equipment, so that the doctor can clearly understand the patient's anatomical position during surgery, improve the safety and success rate of surgery, and effectively reduce the doctor's surgical burden.

[0003] Existing intraoperative navigation methods still have many shortcomings, mainly reflected in the need for doctors to switch back and forth between the display device and the patient, and the difficulty of matching the deformation of human soft tissues during surgery with the preoperative virtual three-dimensional model. With the development of immersive stereoscopic vision technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR), people are committed to applying these new technologies to clinical practice to reduce the interference of surgical navigation systems. Among these technologies, mixed reality technology, as the most efficient and strong real-time interactive new technology, will be the inevitable development trend of immersive intraoperative navigation.

[0004] However, the existing virtual reality has technical bottlenecks that make it difficult to apply in practice. The main reason is that the virtual 3D model of the patient before the operation may not correspond to the actual situation during the operation, which is especially common in soft tissue deformation. For example, in intracranial surgery, factors such as changes in intracranial pressure after the patient's craniotomy can cause soft tissue collapse, resulting in preoperative imaging effects that are inconsistent with the actual situation during the operation. In the existing common intraoperative navigation, this problem needs to be corrected based on the doctor's experience and on-the-spot adaptability, but long-term and high-intensity surgery coupled with such complicated corrections poses a great challenge to the doctor's physical and psychological health.

[0005] Among the existing solutions for adaptive deformation of virtual 3D models, the grid-based tissue deformation model is the main solution. Its advantage is that the model structure is simple and the deformation can be calculated by constructing physical functions between grids to achieve the purpose of correcting the model. Classic algorithms include Loop subdivision method, Butterfly subdivision algorithm, etc. However, this type of method has obvious disadvantages, mainly reflected in the large amount of real-time calculation and poor model robustness, which makes it difficult to actually apply it to surgery.

[0006] With the rapid development of deep learning technology, more and more neural network algorithms have been developed and applied, and have achieved gratifying results in practical applications such as pathological image super-resolution, pathological tissue image recognition, segmentation and detection. However, in the application of soft tissue adaptive deformation, the research and application of deep learning are still insufficient. Compared with existing solutions, deep learning is characterized by strong robustness; the trained network does not require a lot of calculations. It has strong real-time performance. At present, deep learning has achieved excellent results in similar work, such as adaptive deformation of multi-angle three-dimensional models of faces, which can greatly improve the restoration rate and recognition rate of faces.

[0007] Since the adaptive deformation of the model is the main difficulty in whether the model is close to reality in immersive surgery, and deep learning methods have great potential in MR applications, it is of great research significance and broad application prospects to propose a deep learning-based 3D model adaptive method that can adapt to complex medical scenarios. Summary of the invention

[0008] In order to solve the problems in the prior art, the present invention provides a soft tissue adaptive deformation method and system based on mixed reality and 3DGAN to solve the problem of soft tissue three-dimensional model adaptation.

[0009] The soft tissue adaptive deformation method based on mixed reality and 3DGAN of the present invention comprises the following steps:

[0010] S1: Obtain the preoperative global 3D model and convert it into point cloud data;

[0011] S2: During surgery, the depth map data of the patient's local image is obtained in real time and converted into point cloud data;

[0012] S3: sending the point cloud data obtained in step S1 and step S2 into the nested 3DGAN network model to map local deformation to global deformation;

[0013] S4: Convert the deformed point cloud global data into a three-dimensional model.

[0014] In step S3, the nested 3DGAN network model includes a local mapping network and a global detection network. The local mapping network infers the deformation of the global three-dimensional model based on the input local image depth map point cloud data, and the global detection network corrects the globally deformed three-dimensional model output by the local mapping network in this stage based on the preoperative global three-dimensional model point cloud data and the global three-dimensional model point cloud data of the previous stage.

[0015] The present invention is further improved, and before step S1, it also includes a judgment step: judging whether to train the nested 3DGAN network model, if yes, executing, if not, executing step S1, the nested 3DGAN network model training step A includes the following steps:

[0016] A1: obtaining a training set of soft tissue deformation, wherein the training set includes a plurality of different body membrane models;

[0017] A2: Use the training set to train the generator network and discriminator network of the local mapping network until a Nash equilibrium is reached;

[0018] A3: Train the generator network and discriminator network of the global detection network until a Nash equilibrium is reached;

[0019] A4: Training is complete.

[0020] The present invention is further improved, the local mapping network includes a local generator network and a local discriminator network arranged at the rear stage of the local generator network, the processing process of the local generator network is: the local image depth map data after point cloud conversion is processed by N layers of 3D convolution block network, and then processed by the global pooling layer, and finally the deformation of the global three-dimensional model is output, wherein the first N / 2 of the 3D convolution block network adopts void convolution, and the last N / 2 adopts micro-step convolution, and the activation layer of the 3D convolution block network adopts ReLU function as the activation function.

[0021] The present invention is further improved, the input of the local mapping network is a three-dimensional point cloud model generated by the local depth image captured by the mixed reality device during the operation; the output is a preliminary global deformation three-dimensional model inferred in this stage; the loss function of the local mapping network is:

[0022] Local loss =min G max Dlocal V(D local , G)

[0023] =E x～Pdata(x) [logD local (x)]+E z～Pz(z) [1-D local (G local (z))]

[0024] Among them, D local (x) represents the local discriminator, which is used to determine whether the output model is consistent with the phantom data model. The output is a value between 0 and 1. The larger the value, the greater the discriminator believes that the difference between the input and the real image is; min G max D It means that the output of the generator needs to be able to deceive the discriminator, Dlocal represents the discriminator of the local mapping network, G local represents the generator of the local mapping network, z represents the local 3D point cloud model of the current input, and x represents the phantom 3D model of the training set.

[0025] The present invention is further improved, and the processing process of the global detection network is:

[0026] (1) Obtain the global 3D model point cloud data output in the previous stage and the preoperative global 3D model point cloud data, and combine them with the deformation of the global 3D model initially output by the local mapping network in this stage;

[0027] (2) Downsampling the data in step (1) through a downsampling network;

[0028] (3) extracting features through a 3D residual convolutional network including several residual network layers, and performing preliminary fusion output, wherein the pooling activation layer of the 3D residual convolutional network uses a LeakyReLU function as an activation function;

[0029] (4) Outputting the corrected point cloud 3D model after upsampling on the upsampling network;

[0030] (5) Divide the point cloud 3D model into blocks and perform feature fusion output.

[0031] The present invention is further improved, and the processing method of feature fusion in step (5) includes the following steps:

[0032] (501) the preliminary fused model is divided into M blocks, and the global three-dimensional model obtained before surgery is divided into the same block size;

[0033] (502) The preliminary fusion model after the N blocks are divided respectively through the global discriminator of the global detection network;

[0034] (503) The global discriminator outputs the results corresponding to each block. If the output result of the global discriminator is higher than the set threshold, it is directly output. If it is lower than the set threshold, the block data is sent to the block comparison network and the block fusion network of the block feature.

[0035] (504) the block comparison network and the block fusion network search for blocks of the global three-dimensional model obtained before the operation, obtain similarity through inner product, select m blocks with the highest similarity value, fuse the features of the blocks according to the similarity weights, and finally output the adjusted blocks;

[0036] (501) After all blocks are processed by the discriminator, the global deformable 3D point cloud model of this stage is obtained.

[0037] The present invention is further improved. In step (504), the input of the block comparison network is the point cloud block whose output result of the global discriminator is less than the set threshold and the preoperative global point cloud block. The block comparison network includes: a shallow convolution network and a similarity calculation network; the shallow convolution network includes 7 convolution layers, 2 pooling layers and a normalization layer; the calculation formula of the similarity calculation network is:

[0038] h i = <p i / |p i |,q j / |q j |>

[0039] Among them, p and q represent the extracted point cloud features, i represents the point cloud block that needs to be adjusted, and j represents each global point cloud block compared. The calculation method is the inner product, and i and j are positive integers between 1 and M.

[0040] The present invention is further improved, in that the input of the block fusion network is a similarity matrix, and the first m point cloud block features are extracted from it according to the similarity values ​​from large to small. The block fusion network structure includes several layers of down-sampling and up-sampling. The down-sampling adopts void convolution, the up-sampling adopts sub-pixel convolution, the loss function uses perceptual loss function, and the output is an adjusted point cloud block.

[0041] The present invention also provides a system for implementing the above-mentioned soft tissue adaptive deformation method based on mixed reality and 3DGAN, which is characterized by comprising:

[0042] Data acquisition and conversion module: used to obtain the preoperative global 3D model and the patient's local image depth map data during the operation, and convert them into point cloud data;

[0043] Adaptive deformation module: used to feed the acquired point cloud data into the nested 3DGAN network model to realize the mapping of local deformation to global deformation;

[0044] The second data conversion module is used to convert the deformed point cloud global data into a three-dimensional model.

[0045] The nested 3DGAN network model adopted by the adaptive deformation module includes a local mapping network and a global detection network. The local mapping network infers the deformation of the global three-dimensional model based on the input local image depth map point cloud data, and the global detection network corrects the globally deformed three-dimensional model output by the local mapping network in this stage based on the preoperative global three-dimensional model point cloud data and the global three-dimensional model point cloud data of the previous stage.

[0046] The present invention is further improved and further includes a judgment module: used to judge whether to train the nested 3DGAN network model;

[0047] Model training module: used to train the nested 3DGAN network model, the model training module includes:

[0048] An acquisition unit: used for acquiring a training set of soft tissue deformation, wherein the training set includes a plurality of different body membrane models;

[0049] Local mapping network training unit: used to train the generator network and discriminator network of the local mapping network using the training set until a Nash equilibrium is reached;

[0050] Global detection network training unit: used to train the generator network and discriminator network of the global detection network until Nash equilibrium is reached.

[0051] Compared with the prior art, the beneficial effects of the present invention are: it can adapt to complex medical scenarios, and can match the intraoperative soft tissue deformation of the human body with the preoperative virtual three-dimensional model according to the patient's local image during the operation, and generate a three-dimensional model of the deformed soft tissue in the patient's body in real time, so that the doctor can clearly understand the anatomical position of the patient during the operation, improve the safety and success rate of the operation, and effectively reduce the doctor's surgical burden. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 This is a schematic diagram of the structure of the nested 3DGAN network model of the present invention;

[0054] Figure 3 This is a schematic diagram of the structure of the generator part of the local mapping network;

[0055] Figure 4 Schematic diagram of the global detection network structure. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0057] like Figure 1 As shown, the adaptive deformation method of the present invention comprises the following steps:

[0058] Initially, it is determined whether to train the nested 3DGAN (adversarial network) network model. If yes, the adaptive deformation step is performed. If not, the nested 3DGAN network model training step A is performed, including the following steps:

[0059] A1: obtaining a training set of soft tissue deformation, wherein the training set includes a plurality of different body membrane models;

[0060] A2: Use the training set to train the generator network and discriminator network of the local mapping network until a Nash equilibrium is reached;

[0061] A3: Train the generator network and discriminator network of the global detection network until Nash equilibrium is reached;

[0062] A4: Training is complete.

[0063] The adaptive deformation step comprises the following steps:

[0064] S1: Obtain the preoperative global 3D model and convert it into point cloud data;

[0065] S2: During surgery, the depth map data of the patient's local image is obtained in real time and converted into point cloud data;

[0066] S3: sending the point cloud data obtained in step S1 and step S2 into the nested 3DGAN network model to map local deformation to global deformation;

[0067] S4: Convert the deformed point cloud global data into a three-dimensional model.

[0068] In this example, the patient's preoperative 3D model is obtained by 3D reconstruction of preoperative 2D imaging information such as CT or MR, and the patient's intraoperative image is obtained from the doctor's head-mounted mixed reality device, such as Microsoft's HoloLens. The preoperative global 3D model and the intraoperative patient's local image depth map data are converted into point cloud format through the CloudCompare free open source software, and then the point cloud data is sent to the designed nested 3DGAN method to realize the mapping of local deformation to global deformation. Finally, the deformed point cloud global data is converted into a 3D model through CloudCompare. In addition to the point cloud format, the 3D model file format and depth map format need to match the corresponding head-mounted mixed reality device. During the network training process, this example requires multiple sets of different body membrane models as training sets for soft tissue deformation.

[0069] like Figure 2 As shown in the figure, the nested 3DGAN network model in this example is based on DCGAN (Deep ConvolutionalGenerative Adversarial Networks), but its convolution kernel is changed to a three-dimensional convolution kernel to adapt to the input and output point cloud formats.

[0070] The nested 3DGAN network model in this example includes a local mapping network and a global detection network. The function of the local mapping network is to infer the deformation of the global three-dimensional model based on the input local point cloud model. The function of the global detection network is to correct the globally deformed three-dimensional model output by the local mapping network in this stage based on the preoperative global three-dimensional model and the global three-dimensional model of the previous stage, to ensure the consistency and authenticity of its model changes.

[0071] In this method, the discriminators of the two nested networks both use the patchGAN method, but the difference is that the global detection network uses the LeakyReLU function as the activation function, while the local mapping network uses the ReLU function as the activation function. The nesting in this method is reflected in that the global detection network will call the generator of the local detection network, correct the deformation of the global three-dimensional model output by the generator of the local detection network, and output the three-dimensional model after global deformation, so that in the process of real-time acquisition of local image data by the head-mounted mixed reality device, real-time deformed data is generated, so that the deformation of human soft tissue during surgery matches the virtual three-dimensional model before surgery, and a three-dimensional model of the soft tissue deformed in the patient's body is generated in real time, so that the doctor can clearly understand the anatomical position of the patient during surgery, improve the safety and success rate of surgery, and effectively reduce the burden of surgery on the doctor.

[0072] like Figure 3 As shown in the figure, the processing process of the local generator network is as follows: the local image depth map data after point cloud conversion is processed by N layers of 3D convolution block network, and then processed by the global pooling layer, and finally the deformation of the global three-dimensional model is output, wherein the first N / 2 of the 3D convolution block network use void convolution, and the last N / 2 use micro-step convolution.

[0073] Specifically, the input of the local mapping network in this example is: the 3D point cloud model generated by the local depth image captured by the mixed reality device during surgery. Output: the preliminary global deformable 3D model inferred in this stage.

[0074] The loss function is:

[0075] Local loss =min G max Dlocal V(D local , G)

[0076] =E x～Pdata(x) [logD local (x)]+E z～Pz(z) [1-D local (G local (z))]

[0077] Among them, D local(x) represents a local discriminator, which is used to determine whether the output model is consistent with the phantom data model. In this method, the output of the discriminator of the generative adversarial network is a value between 0 and 1. The larger the value, the greater the discriminator believes that the input is different from the real image; min G max D It means that the output of the generator needs to be able to deceive the discriminator, that is, to minimize its output, and the discriminator needs to distinguish between real and generated fake images by learning the real data distribution, that is, to maximize the output. The final result of the training is that the discriminator and the generator reach Nash equilibrium. local represents the discriminator of the local mapping network, G local represents the generator of the local mapping network, z represents the local 3D point cloud model of the current input, and x represents the phantom 3D model of the training set. During the training process, D local Will participate in the algorithm and update the parameters. After the training is completed, only G local part.

[0078] like Figure 4 As shown in the figure, the three-dimensional model with high-precision texture generated by the GAN method has a good visual effect, but because the generator network tends to deceive the discriminator network, the high-frequency texture of the generated model tends to the model in the training data set, resulting in a large deviation in the detail texture of the generated model and the target point cloud model in actual use; this method designs a new global monitoring network, which uses feature discrimination, feature fusion and other methods to correct the output of the local mapping network to ensure the accuracy of the output soft tissue point cloud model.

[0079] The global detection network input in this case is: the global 3D model obtained before surgery, the output of the global network in the previous stage, and the preliminary global deformable 3D model from the local mapping network in this stage.

[0080] Output: The global deformable 3D model inferred in this stage.

[0081] The loss function is:

[0082] Global loss =min G max Dglobal V(D global , G)

[0083] =E x～Pdata(x) [logD global (x)]+E z～Pz(z) [1-D global (G local (z)+x mark *Mark+x global *Global)]

[0084] Among them, global means introducing the global model generated by preoperative medical imaging data, x global represents the weight of the preoperative global model, mark represents the final output global model of the previous stage, and x mark It represents the weight of the global model finally outputted in the previous stage. The purpose of introducing the preoperative global model is to ensure the consistency and authenticity of the preoperative and intraoperative models, and the purpose of introducing the global model finally outputted in the previous stage is to ensure the continuity of deformation. This method needs to find the balance point between deformation and consistency through multiple adjustments of the two weight values, and in general, x global The value of needs to increase as the number of stages increases. global represents the discriminator of the global detection network, G local (z) represents the preliminary local 3D point cloud model of the current input, and x represents the 3D model of the phantom in the training set. The generator part of the global detection network uses a residual network design to prevent overfitting because the input data is large.

[0085] The intermediate process of this example is:

[0086] The preliminary global deformable 3D model of the local mapping network in this stage and the output of the global network in the previous stage are feature fused and output through the designed 3D residual convolutional network to ensure the output consistency between each stage.

[0087] Specifically, the processing process of the global detection network is:

[0088] (1) Obtain the global 3D model point cloud data output in the previous stage and the preoperative global 3D model point cloud data, and combine them with the deformation of the global 3D model initially output by the local mapping network in this stage;

[0089] (2) Downsampling the data in step (1) through a downsampling network;

[0090] (3) Feature extraction is performed through a 3D residual convolutional network including several residual network layers, and preliminary fusion output is performed;

[0091] (4) Outputting the corrected point cloud 3D model after upsampling on the upsampling network;

[0092] (5) Divide the point cloud 3D model into blocks and perform feature fusion output.

[0093] As an embodiment of the present invention, the processing method of feature fusion in this example is:

[0094] The preliminary fused model is divided into M blocks, and the size of each block is divided according to the required model output accuracy requirements (the higher the detail texture and accuracy requirements, the smaller the size of each block, but it is not recommended to be less than 7*7*7, because too small point cloud blocks will lose globality, that is, the relationship with the surrounding point clouds), and the global 3D model obtained before surgery is divided into the same block size; the preliminary fused model after M blocks is divided into discriminators of the global detection network, and the discriminator will output a prediction value. If the prediction value is higher than the set threshold, this block is considered to be the final output block. If it is lower than the threshold, it needs to be sent to the block feature comparison and fusion module of the global detection network; the block comparison and fusion network will search for the blocks of the global 3D model obtained before surgery, obtain the similarity through the inner product, select the m blocks with the highest similarity value, perform block feature fusion according to the similarity weight, and finally output the adjusted blocks; after all blocks are processed by the discriminator, the global deformable 3D point cloud model of this stage can be obtained. After all blocks are processed by the discriminator, the global deformable 3D point cloud model of this stage can be obtained.

[0095] The loss function of the discriminator network in this example is:

[0096] Global loss =min G max D V(D pre , G)

[0097] =E x～Pdata(x) [logD(x)]+E z～Pz(z) [1-D(G pre (z)+x mark *Mark)]

[0098] Among them, G pre (z) represents the output from the local mapping network in this stage, mark represents the global model of the final output of the previous stage, and x mark It represents the weight of the global model finally outputted in the previous stage. The output of the global network in the previous stage is introduced to ensure the continuity of deformation. x represents the phantom 3D model of the training set.

[0099] The output of the global discriminator is a numerical value between 0 and 1. The larger the value, the higher the confidence of the block. In general, the threshold is designed to be 0.8, which can be changed dynamically according to actual needs.

[0100] The input of the block comparison network in this example is the point cloud block with insufficient confidence and the preoperative global point cloud block. The block comparison network includes: a shallow convolution network and a similarity calculation network; the shallow convolution network includes 7 convolution layers, 2 pooling layers and a normalization layer; the calculation formula of the similarity calculation network is:

[0101] h i= <p i / |p i |,q j / |q j |>

[0102] Among them, h i is the input of the block fusion network, p, q represent the extracted point cloud features, i represents the point cloud block to be adjusted, j represents each global point cloud block compared, and the calculation method is the inner product. i, j are positive integers between 1 and M.

[0103] The input of the block fusion network in this example is the similarity matrix, and the first m point cloud block features are extracted from it according to the similarity value from large to small (m ≥ 5, increased according to demand, the smoother the texture details, the larger m). The block fusion network structure of this example refers to the U-net network (deep learning segmentation network) structure, which consists of 8 layers of down-sampling and up-sampling. The down-sampling adopts void convolution, the up-sampling adopts Sub-pixel convolution, and the loss function uses the perceptual loss function; the output is the adjusted point cloud block.

[0104] Through the global detection network of the present invention, the detail texture deviation between the model generated by the present invention in actual surgery and the target point cloud model is greatly reduced, and the generated accuracy is high, meeting the needs of doctors in complex surgical scenarios.

[0105] The specific implementation modes described above are preferred implementation modes of the present invention, and are not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to the specific implementation modes, and all equivalent changes made according to the present invention are within the protection scope of the present invention.

Claims

1. A soft tissue adaptive deformation method based on mixed reality and 3DGAN, characterized by: The steps include: S1: Obtain the preoperative global 3D model and convert it into point cloud data; S2: During surgery, the depth map data of the patient's local image is obtained in real time and converted into point cloud data; S3: sending the point cloud data obtained in step S1 and step S2 into the nested 3DGAN network model to map local deformation to global deformation; S4: Convert the deformed point cloud global data into a three-dimensional model. In step S3, the nested 3DGAN network model includes a local mapping network and a global detection network. The local mapping network infers the deformation of the global three-dimensional model based on the input local image depth map point cloud data, and the global detection network corrects the globally deformed three-dimensional model output by the local mapping network in this stage based on the preoperative global three-dimensional model point cloud data and the global three-dimensional model point cloud data of the previous stage.

2. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 1, characterized in that: Before step S1, a judgment step is also included: judging whether to train the nested 3DGAN network model, if yes, executing step S1, if not, executing step A of training the nested 3DGAN network model, including the following steps: A1: obtaining a training set of soft tissue deformation, wherein the training set includes a plurality of different body membrane models; A2: Use the training set to train the generator network and discriminator network of the local mapping network until a Nash equilibrium is reached; A3: Train the generator network and discriminator network of the global detection network until Nash equilibrium is reached; A4: Training is complete.

3. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 1 or 2, characterized in that: The local mapping network includes a local generator network and a local discriminator network arranged at the rear stage of the local generator network. The processing process of the local generator network is as follows: the local image depth map data after point cloud conversion is processed by N layers of 3D convolution block network, and then processed by the global pooling layer, and finally the deformation of the global three-dimensional model is output, wherein the first N / 2 of the 3D convolution block network adopts void convolution, and the last N / 2 adopts micro-step convolution, and the activation layer of the 3D convolution block network adopts ReLU function as the activation function.

4. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 1 or 2, characterized in that: The input of the local mapping network is the 3D point cloud model generated by the local depth image captured by the mixed reality device during surgery; the output is the preliminary global deformation 3D model inferred in this stage; the loss function of the local mapping network is: Local loss =min c max Dlocal V(D local ,G) =E x～Pdata(x) [IogD local (X)]+E z～Pz(z) [1-D local (G local (Z))] Among them, D local (x) represents the local discriminator, which is used to determine whether the output model is consistent with the phantom data model. The output is a value between 0 and 1. The larger the value, the greater the discriminator believes that the difference between the input and the real image is; min G max D It means that the output of the generator needs to be able to deceive the discriminator, D local represents the discriminator of the local mapping network, G local represents the generator of the local mapping network, z represents the local 3D point cloud model of the current input, and x represents the phantom 3D model of the training set.

5. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 1 or 2, characterized in that: The processing process of the global detection network is: (1) Obtain the global 3D model point cloud data output in the previous stage and the preoperative global 3D model point cloud data, and combine them with the deformation of the global 3D model initially output by the local mapping network in this stage; (2) Downsampling the data in step (1) through a downsampling network; (3) extracting features through a 3D residual convolutional network including several residual network layers, and performing preliminary fusion output, wherein the pooling activation layer of the 3D residual convolutional network uses a LeakyReLU function as an activation function; (4) Outputting the corrected point cloud 3D model after upsampling on the upsampling network; (5) Divide the point cloud 3D model into blocks and perform feature fusion output.

6. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 5, characterized in that: Step (5) The feature fusion processing method includes the following steps: (501) The preliminary fused model is divided into M blocks, and the global three-dimensional model obtained before surgery is divided into the same block size; (502) The preliminary fusion model after the N blocks are divided respectively through the global discriminator of the global detection network; (503) The global discriminator outputs the results corresponding to each block. If the output result of the global discriminator is higher than the set threshold, it is directly output. If it is lower than the set threshold, the block data is sent to the block comparison network and the block fusion network of the block feature. (504) the block comparison network and the block fusion network search for blocks of the global three-dimensional model obtained before the operation, obtain similarity through inner product, select m blocks with the highest similarity value, fuse the features of the blocks according to the similarity weight, and finally output the adjusted blocks; (505) After all blocks are processed by the discriminator, the global deformable 3D point cloud model of this stage is obtained.

7. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 6, characterized in that: In step (504), the input of the block comparison network is the point cloud block whose output result of the global discriminator is less than the set threshold and the preoperative global point cloud block, and the block comparison network includes: a shallow convolution network and a similarity calculation network; the shallow convolution network includes 7 convolution layers, 2 pooling layers and a normalization layer; the calculation formula of the similarity calculation network is: h i =<p i / |p i |,q j / |q j |> Among them, p and q represent the extracted point cloud features, i represents the point cloud block that needs to be adjusted, and j represents each global point cloud block compared. The calculation method is the inner product, and i and j are positive integers between 1 and M.

8. The method for adaptive deformation of soft tissue based on mixed reality and 3DGAN according to claim 7, characterized in that: The block fusion network input is a similarity matrix, and the first m point cloud block features are extracted from it according to the similarity value from large to small. The block fusion network structure includes several layers of down-sampling and up-sampling. Down-sampling uses void convolution, up-sampling uses sub-pixel convolution, the loss function uses perceptual loss function, and the output is the adjusted point cloud block.

9. A system for implementing the soft tissue adaptive deformation method based on mixed reality and 3DGAN according to any one of claims 1 to 8, characterized in that: include: Data acquisition and conversion module: used to obtain the preoperative global 3D model and the patient's local image depth map data during the operation, and convert them into point cloud data; Adaptive deformation module: used to feed the acquired point cloud data into the nested 3DGAN network model to realize the mapping of local deformation to global deformation; The second data conversion module is used to convert the deformed point cloud global data into a three-dimensional model. The nested 3DGAN network model adopted by the adaptive deformation module includes a local mapping network and a global detection network. The local mapping network infers the deformation of the global three-dimensional model based on the input local image depth map point cloud data, and the global detection network corrects the globally deformed three-dimensional model output by the local mapping network in this stage based on the preoperative global three-dimensional model point cloud data and the global three-dimensional model point cloud data of the previous stage.

10. The system according to claim 9, characterized in that: It also includes a judgment module: used to judge whether to train the nested 3DGAN network model; Model training module: used to train the nested 3DGAN network model, the model training module includes: An acquisition unit: used for acquiring a training set of soft tissue deformation, wherein the training set includes a plurality of different body membrane models; Local mapping network training unit: used to train the generator network and discriminator network of the local mapping network using the training set until a Nash equilibrium is reached; Global detection network training unit: used to train the generator network and discriminator network of the global detection network until Nash equilibrium is reached.

Images