Full-size thoracic cavity 3D model reconstruction method based on single-view depth image

Abstract

The invention is suitable for the technical field of auxiliary medical treatment, and provides a full-size thoracic cavity 3D model reconstruction method based on a single-view depth image, and the method comprises the following steps: generating shape priori through Gaussian noise disturbance postoperative thoracic cavity point cloud; an AFF module is adopted to adaptively fuse the depth image features and the shape prior features; inputting the fusion features into a graph-based convolutional neural network, and outputting vertex coordinates of the reconstructed thoracic cavity three-dimensional model; on the basis of global feature information, predicting relative displacement of a root node of the thoracic cavity model in a world coordinate system to realize model space alignment; and performing iterative optimization. According to the method, based on the neural network model trained by the constructed chest simulation data set, a full-size chest three-dimensional grid can be obtained, so that the volume difference of the breasts on the left side and the right side can be accurately calculated, the volume needing to be filled after a single-side breast is cut off is accurately estimated, and then it is ensured that the breasts of a postoperative patient are symmetric; and the secondary repair operation rate is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of assistive medical technology, and in particular relates to a method for reconstructing a full-size 3D model of the thoracic cavity based on a single-view depth image. Background Technology

[0002] With increased health awareness and advancements in clinical imaging technology, the early detection rate of breast cancer has approached 90%. Simultaneously, the prognosis for breast cancer patients continues to improve, with a 5-year survival rate exceeding 90% and a 10-year survival rate surpassing 80%. Currently, breast-conserving surgery, which removes breast tumors while preserving breast shape, is becoming increasingly mainstream due to its significant improvement in postoperative quality of life. Examples include traditional breast-conserving surgery, breast reduction surgery, and breast reconstruction surgery using autologous or implant tissue.

[0003] Existing methods for assessing bilateral breast volume difference mainly fall into two categories: objective measurement and subjective visual assessment. Objective measurement includes mechanical measurement, magnetic resonance imaging (MRI), and 2D or 3D shape imaging. However, mechanical measurement has drawbacks such as contact, inconvenience, and difficulty in accurately representing non-rigid curved surfaces; MRI relies on large equipment and complex image reconstruction; 2D and 3D imaging are mostly performed while the patient is standing, making the operation complex and time-consuming. Therefore, existing objective measurement methods are not suitable for use during surgery. Consequently, subjective visual assessment of bilateral breast volume difference is now primarily used during surgery. This method relies on the surgeon's visual experience, is susceptible to individual differences and visual errors, and only achieves about 70% postoperative symmetry, making it difficult to provide accurate and reliable assessment results.

[0004] Therefore, a method is needed to assess the volume difference between the two sides of the breast, reconstruct a full-size 3D model of the thoracic cavity based on single-view depth images taken during surgery, determine the volume that needs to be filled on the affected side, and thus guide the surgery to ensure postoperative breast symmetry. Summary of the Invention

[0005] The purpose of this invention is to provide a method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image, aiming to solve the problems mentioned in the background art.

[0006] The present invention is implemented as follows: a method for reconstructing a full-size 3D model of the thoracic cavity based on a single-view depth image, comprising the following steps:

[0007] Step 1: Generate shape priors by perturbing the postoperative thoracic cavity point cloud with Gaussian noise;

[0008] Step 2: Adaptively fuse depth image features and shape prior features using the AFF module (Adaptive Feature Fusion Module);

[0009] Step 3: 3D mesh prediction;

[0010] The fused features are input into a graph convolutional neural network, which outputs the vertex coordinates of the reconstructed 3D model of the thoracic cavity.

[0011] Step 4: Root node displacement estimation;

[0012] Based on global feature information, the relative displacement of the root node of the thoracic cavity model in the world coordinate system is predicted to achieve model space alignment.

[0013] Step 5: Iterative optimization;

[0014] The initial prediction results are introduced into the noise and used as a new shape prior. Multiple rounds of prediction and fusion are performed to gradually improve the reconstruction accuracy.

[0015] A further technical solution, wherein step 1 includes the following specific steps:

[0016] A local full-size 3D model of the thoracic cavity is generated based on the obtained depth image. This process uses the function f. LCL This is expressed as formula (1);

[0017] x = f LCL (I D (1)

[0018] Among them, I D For a depth image, x = (V, E) represents a local full-size chest model, which is composed of N... v The set of vertices V = {v1, v2, ..., v...} Nv} and connecting vertex N e The set of edges of a triangle The resulting 3D topological graphics;

[0019] Based on formula (1), a shape prior x is given, which contains a rough human body shape. Then formula (1) becomes:

[0020] x = f LCL (x,I D (2)

[0021] The shape priors x with different confidence levels are generated by adding noise, and the specific process is shown in formula (3);

[0022]

[0023] Where ε is Gaussian noise; N(0,1) is a standard normal distribution; t∈{1,2,…,T} is the fuzziness intensity of the current shape prior, which is randomly selected during training; T is set to 1000; αt =1-β t ,β t This represents the fuzzy intensity coefficient at step t, which increases with t, and its value ranges from [5 × 10^t]. -5 5×10 -4 [Achieved linearly;]

[0024] Further, formula (2) is visualized as formula (4):

[0025]

[0026] This expression can be interpreted as using depth image I D The observed shape f D (I D For a rough human body shape f from the shape prior x T (x) is corrected to obtain a chest cavity model that is closer to the real human body; in the above formula Indicates the correction method.

[0027] A further technical solution, step 2 includes the following steps:

[0028] Considering that the credibility of features from different information sources varies for different parts, an AFF module is proposed to implement the correction method in formula (4) through feature fusion.

[0029] First, the similarity factor S between the shape prior and the depth image information is obtained by calculating the inner product. R The difference between shape prior and true observation is determined, and then it is input into the modified weight mapping layer to obtain the modified weight. Finally, the adaptive fusion of the two features is achieved using formula (5).

[0030]

[0031] In the formula, F x F represents the shape characteristics of the fused thoracic cavity. x and F D These are shape features extracted from shape priors and depth images, respectively, both of which are N-type features describing the shape of different parts of the thoracic cavity. x A set of N eigenvectors x It is set to 52; w is a set of weight factors located in the interval [0,1], where the superscript i indicates the i-th element in the set;

[0032] The Projetion Layer has a weight w p and bias b p An optimizable mapping layer is represented by Equation (6);

[0033]

[0034] Among them, S′ R This represents the optimized similarity factor;

[0035] To ensure that w always lies within the interval [0,1], formulas (7) and (8) are used to... Perform normalization processing;

[0036]

[0037] in, Make S′ R To make it positive, max(·) means to take the maximum value from the set.

[0038] A further technical solution involves using a graph convolution-based Thoracic Decoder neural network in step 3 to convert F... x The result is transformed into a set of local coordinate positions of the vertices, and thus formula (4) is further expressed as formula (9);

[0039] x = f TD (f AFF (f T (x),f D (I D ))) (9)

[0040] Among them, f AFF Indicates the AFF module, f TD This indicates Thoracic Decoder.

[0041] In a further technical solution, step 4 includes the following specific steps:

[0042] To obtain the position r of the thoracic root node in the world coordinate system g The coarse offset of the thoracic cavity in the basic world coordinate system is extracted from the depth image, and then further corrected using the shape information of the thoracic cavity, thereby obtaining a more accurate world coordinate system position. This process can be represented by formula (10).

[0043] r g =f r (f Dr (F D )+f xr (F x )) (10)f Dr Described as a feature-based global average pooling layer, f r For the Root Estimator neural network, and f xr This is only a global average calculation.

[0044] The LCL neural network model proposed based on the above concepts includes the following training loss function terms;

[0045] Local Consistency Loss is used to describe the difference between the predicted thoracic cavity 3D model x and the true value x, and is specifically expressed as Equation (11);

[0046] L LC =sml1(x,x) (11)

[0047] Where sml1(·) represents the Smooth loss error function;

[0048] The Deep Features Consistentloss is shown in Equation (12).

[0049]

[0050] Where mse(·) represents the Mean Squared Error function; These are dynamic weighting coefficients used to describe the fused feature F. x The higher the reliability of F, the greater the reliability. x The more credible; where α c The constant weighting factor is set to 10; δ l =L LC For dynamic weighting factors;

[0051] The shape features consistent loss is shown in Equation (13).

[0052]

[0053] in, Represents the dynamic time weighting coefficient; ζ t As a constant weighting factor, the smaller t decreases, the closer the true shape value is to the prior shape value;

[0054] Root Consistency Loss describes the predicted root node position. Its truth value r g The differences between them are shown in formula (14);

[0055]

[0056] Global Consistency Loss is used to describe the position x of the thoracic cavity model in the predicted world coordinate system. g x between its truth value g The differences are shown in formula (15);

[0057] L GC =sml1(x g ,x g (15)

[0058] Where x g and x g Obtained through formulas (16) and (17);

[0059]

[0060] x g =x+r g (17)

[0061] Consistent Details Loss is achieved by obtaining details from x. g and x g The differences between the rendered depth images further constrain the shape details of the prediction model, and this process is expressed as formula (18);

[0062] L VGG =vgg(f r (x g ),f r (x g (18)

[0063] Among them, f r (·) represents the renderer function, and vgg(·) represents the VGG loss function;

[0064] Finally, the above loss terms are integrated according to formula (19) to obtain the integrated loss function for model training;

[0065] L=ω f ×(ω g ×L GC +ω v ×L VGG )+ω r ×L RC +ω l ×L LC +L DFC +L SFC (19)

[0066] Where, ω f ω g ωv ω r and ω l These are the weight hyperparameters, set to 10, 0.7, 0.3, 5, and 7 respectively.

[0067] A further technical solution is that, in step 5, considering the significant difference between the initial shape prior and the actual human body shape, an iterative strategy is adopted. This strategy adds noise to each prediction result and uses it as a new shape prior in the next iteration. As the number of iterations increases, the added noise gradually decreases, and the shape prior gradually approaches the actual human body shape.

[0068] This invention provides a method for reconstructing a full-size 3D model of the chest cavity based on single-view depth images. This method only requires the deployment of a single-view depth camera to obtain real images of the patient's breast area during surgery, without interfering with the surgical process and with low camera cost. Based on the constructed chest simulation dataset, a neural network model trained can fill in the missing information in local areas caused by the absorption of infrared beams emitted by the sensor by dark skin, resulting in a full-size three-dimensional mesh of the chest. This allows for accurate calculation of the volume difference between the left and right breasts, and a more accurate estimation of the volume that needs to be filled after the removal of one breast, thereby ensuring postoperative breast symmetry and effectively reducing the rate of secondary repair surgery. Attached Figure Description

[0069] Figure 1 Detailed diagram of the adaptive feature fusion module structure;

[0070] Figure 2 The diagram shows the LCL model structure and training details.

[0071] Figure 3 This is a diagram illustrating the prediction process of the LCL model.

[0072] Figure 4 A diagram of the created breast model;

[0073] Figure 5 This is a diagram of the modified human body model;

[0074] Figure 6 A sample image of the generated human chest simulation data;

[0075] Figure 7 The flowchart illustrates a method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image, as provided in an embodiment of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0077] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0078] like Figure 7 As shown, an embodiment of the present invention provides a method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image, comprising the following steps:

[0079] Step 1: Generate shape priors by perturbing the postoperative thoracic cavity point cloud with Gaussian noise;

[0080] To obtain richer navigation information, a local full-size 3D model of the thoracic cavity is generated based on the acquired depth image. This process can be performed using the function f. LCL This is expressed as formula (1);

[0081] x = f LCL (I D (1)

[0082] Among them, I D For a depth image, x = (V, E) represents a local full-size chest model, which is composed of N... v The set of vertices V = {v1, v2, ..., v...} Nv} and connecting vertex N e The set of edges of a triangle The resulting 3D topological graph. In this method, the positions of the vertices change with the shape of the human body, while the topological connections described by the edges and the number of vertices are fixed. Therefore, formula (1) essentially only calculates the position information of the vertices. Nevertheless, the depth image I D It only contains information about the anterior surface of the thoracic cavity, while the lateral and posterior surfaces are obscured, making it quite difficult to directly apply formula (1). However, the geometry of the human thoracic cavity is similar, so a shape prior x is given to formula (1), which contains a rough human shape. At this point, formula (1) becomes:

[0083] x = f LCL (x,I D (2)

[0084] The shape priors x with different confidence levels are generated by adding noise, and the specific process is shown in formula (3).

[0085]

[0086] Where ε is Gaussian noise; N(0,1) is a standard normal distribution; t∈{1,2,…,T} is the fuzziness intensity of the current shape prior, which is randomly selected during training; in this method, T is set to 1000; α t =1-β t ,β t This represents the fuzzy intensity coefficient at step t, which increases with t, and its value ranges from [5 × 10^t]. -5 5×10 -4 Linear acquisition is used to obtain,

[0087] In the implementation process, formula (2) can be further visualized as formula (4):

[0088]

[0089] This formula can be interpreted as using depth image I D The observed shape f D (I D For a rough human body shape f from the shape prior x T (x) is modified to obtain a chest cavity model that more closely resembles a real human body. In the above formula... Indicates the correction method.

[0090] Step 2: Adaptively fuse depth image features and shape prior features using the AFF module;

[0091] Considering that the credibility of features from different information sources varies for different body parts, for example, depth image features are usually more reliable for features on the anterior surface of the thoracic cavity, while shape prior features are more reliable for features in other areas, the AFF module is proposed to implement the correction method in formula (4) through feature fusion.

[0092] Figure 1 The structural details of the AFF module are shown. First, the similarity factor S between the shape prior and the depth image information is obtained by calculating the inner product. R The difference between the shape prior and the true observation is determined, and then the difference is input into the modified weight mapping layer to obtain the modified weight. Finally, the adaptive fusion of the two features is achieved using formula (5).

[0093]

[0094] In the formula, F x F represents the shape characteristics of the fused thoracic cavity. x and F D These are shape features extracted from shape priors and depth images, respectively, both of which are N-type features describing the shape of different parts of the thoracic cavity. x A set of N eigenvectors, in the experiments of this methodx It is set to 52; w is a set of weight factors located in the interval [0,1], and the superscript i indicates the i-th element in the corresponding set.

[0095] Figure 1 In the Projetion Layer, there is a weight w. p and bias b p An optimizable mapping layer can be represented by Equation (6).

[0096]

[0097] Among them, S′ R This represents the optimized similarity factor;

[0098] To ensure that w always lies within the interval [0,1], formulas (7) and (8) are used to... Normalization is performed.

[0099]

[0100] in, Make S′ R To make it positive, max(·) means to take the maximum value from the set.

[0101] Step 3: 3D Mesh Prediction

[0102] This method uses a graph convolution-based Thoracic Decoder neural network to convert F... x The result is transformed into a set of local coordinate positions of the vertices, and thus formula (4) is further expressed as formula (9).

[0103] x = f TD (f AFF (f T (x),f D (I D ))) (9)

[0104] Among them, f AFF Indicates the AFF module, f TD This indicates Thoracic Decoder.

[0105] Step 4: Root node displacement estimation;

[0106] To obtain the position r of the thoracic root node in the world coordinate system g For mixed reality navigation used in surgery, this method extracts a coarse offset of the thoracic cavity in the base world coordinate system from a depth image, and then further corrects it using thoracic cavity shape information, thereby obtaining a more accurate world coordinate system position. This process can be represented by formula (10).

[0107] r g =f r (f Dr (F D )+f xr (F x (10)

[0108] In specific implementation, f Dr Described as a feature-based global average pooling layer, f r For the Root Estimator neural network, and f xr This is only a global average calculation.

[0109] Figure 2 The structural details and training process of the LCL neural network model proposed based on the above concepts are shown. As shown in the figure, the model includes the following training loss function terms.

[0110] Local Consistency Loss is used to describe the difference between the predicted 3D model x of the thoracic cavity and the true value x, and can be expressed as Equation (11).

[0111] L LC =sml1(x,x) (11)

[0112] Where sml1(·) represents the Smooth loss error function.

[0113] The depth features consistent loss makes the shape features extracted from the depth image as similar as possible to the features after AFF fusion, as shown in formula (12).

[0114]

[0115] Where mse(·) represents the Mean Squared Error function. These are dynamic weighting coefficients, which describe F x The higher the reliability of F, the greater the reliability. x The more credible, the better. Where α... c The constant weighting factor is set to 10 in this experiment, δ l =L LC This is a dynamic weighting factor.

[0116] The shape features consistent loss is shown in Equation (13).

[0117]

[0118] in ζ represents the dynamic time weighting coefficient. t As the weighting factor t decreases, the true shape value becomes closer to the shape prior. In this case, Shape Features Consistentloss makes the shape prior features as similar as possible to the features fused by AFF.

[0119] Both Depth Features Consistent loss and Shape Features Consistent loss ensure the consistency of shape information features and depth image features in style, thus guaranteeing the logical rationality of the weighted addition operation between the two in AFF.

[0120] Root Consistency Loss describes the predicted root node position. Its truth value r g The differences between them are shown in formula (14);

[0121]

[0122] Global Consistency Loss is used to describe the position x of the thoracic cavity model in the predicted world coordinate system. g x between its truth value g The differences are shown in formula (15).

[0123] L GC =sml1(x g ,x g (15)

[0124] Where x g and x g Obtained through formulas (16) and (17).

[0125]

[0126] x g =x+r g (17)

[0127] Consistent Details Loss is achieved by obtaining details from x. g and x g The differences between the rendered depth images further constrain the shape details of the prediction model, a process that can be expressed as formula (18).

[0128] LVGG =vgg(f r (x g ),f r (x g (18)

[0129] Among them, f r (·) is the renderer function, and vgg(·) represents the VGG loss function. In the specific implementation, the weights of each feature layer in this method were adjusted to 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 16 respectively.

[0130] Finally, the above loss terms are integrated according to formula (19) to obtain the integrated loss function for model training.

[0131] L=ω f ×(ω g ×L GC +ω v ×L VGG )+ω r ×L RC +ω l ×L LC +L DFC +L SFC (19)

[0132] Where, ω f ω g ω v ω r and ω l These are the weight hyperparameters, set to 10, 0.7, 0.3, 5, and 7 respectively.

[0133] Step 5: Iterative optimization;

[0134] Figure 3 The testing process of the LCL model is demonstrated. Considering the significant difference between the initial shape prior and the actual human body shape, an iterative strategy is employed. This strategy introduces noise into each prediction result, which is then used as a new shape prior in the next iteration. As the number of iterations increases, the added noise gradually decreases, thus the shape prior gradually approaches the actual human body shape. Simultaneously, due to the addition of noise during iteration, only a portion of the shape information of the predicted values ​​is retained. Therefore, even if model fluctuations cause significant errors between the predicted and true values ​​during iteration, this will not have a major impact on the next iteration, thus ensuring the smoothness of the iteration process.

[0135] In a preferred embodiment of the present invention, due to the high complexity and difficulty of breast reconstruction surgery, hospitals can only provide a limited number of medical records, and many patients, considering privacy, do not allow images to be collected during the surgery. This makes it very difficult to collect a sufficient amount of image data for model training, and it is also very difficult to accurately annotate a full-size 3D model of the chest cavity using single-view point clouds. Therefore, this method constructs a simulation dataset to train the model. To reduce the impact of differences between the simulation data and the real data domain on the model output, a Zero-shot Domain Adaptation module is embedded in the iteration process. This module can correct the gap between the prediction results and the observations, further penetrating the shape prior and affecting subsequent iterations.

[0136] The final predicted full-size 3D model of the chest cavity was further analyzed to generate information such as the shape and volume differences between the two sides of the chest for surgical navigation.

[0137] As a preferred embodiment of the present invention, the Zero-shot Domain Adaptation module first finds the vertex v in the world coordinate system. g Point p in the recent perceptual point cloud v The distance d between the two is then used. Figure 3 The two methods shown make v g Close to p v .

[0138] The first method is called Forward Fitting, as shown in formulas (20) and (21).

[0139] δ=p v -v g (20)

[0140] v' g =δ z +v g (twenty one)

[0141] Where δ is v g Point to p v The vector, δ z This is the 3D offset vector after setting the horizontal and vertical offsets of δ to zero.

[0142] The second method is called Normal Smoothing, as shown in formulas (22) and (23).

[0143]

[0144] in, d represents the mean of d, tanh(·) represents the hyperbolic tangent function, and · represents the inner product operation. This indicates that the point cloud surface is at point P. v The normal vector at Δ n This indicates the correction step size, which adaptively adjusts the correction magnitude based on the Euclidean distance between each point and its target position.

[0145] The first method primarily acts on the anterior surface of the thoracic cavity, while the second method primarily acts on the sides of the thoracic cavity.

[0146] In a preferred embodiment of the present invention, the process for constructing the simulation dataset is as follows:

[0147] Using the SMPL human body model as a base template, a full-size model reconstruction simulation dataset with consistent 3D shape style annotations was established. Different body postures were generated by changing the shape parameters of the SMPL model. However, due to the asymmetry of the chest caused by the removal of the mammary glands during surgery, the breast tissue slipped to both sides under the influence of gravity, resulting in the collapse of the anterior surface of the chest and the outward extension of the sides of the chest, producing shape changes that were difficult for the original model to accurately simulate. Therefore, the human body model was modified through the following steps.

[0148] First, 659 vertices located on the chest were manually selected to construct the chest model (e.g., Figure 4 (as shown in a), and place it in a custom coordinate system for easy subsequent processing. The coordinate system takes O in formula (24) as the origin, the direction of the human body from left to right as the positive x-axis, the direction of the human body from the feet to the head as the positive y-axis, and the direction of the human body facing forward as the positive z-axis.

[0149]

[0150] in These represent the mean values ​​of all vertices of the chest model in the x, y, and z directions, respectively.

[0151] Next, the area located on the negative x-axis is considered the left breast, and the rest is considered the right breast. After mirroring the left breast along the yoz plane to the positive x-axis, formula (25) is used to process the left and right breasts respectively.

[0152]

[0153] Where, δ d The expansion factor is calculated as shown in formula (26); and Factors are used to define the regions of shape change in the x and y directions. As shown in formula (27), It has a similar form.

[0154] δ d =σ d ×k d ×v i (26)

[0155] Where, σ d For the expansion symbol, when its vertex v i When located on the affected side, σ d =-1, when located on the healthy side, σ d =1. k d ~[0.01,0.03] is the expansion size factor, and its value is the same for the same breast on the same side of the same human body.

[0156]

[0157] Where, x i For vertex v i x-coordinate value, x min and x max The minimum and maximum x-coordinates of all vertices on one side of the chest are given by v. i When located on the affected side x min =0; when v i When located on the healthy side, x min =0.7×x max ;and In calculation At that time, y min It represents the minimum y-coordinate of all vertices on one side of the chest.

[0158] The modified breast model, as shown Figure 4 As shown in b.

[0159] Then, the chest model is transformed back to the original coordinate system using the center point O, thus restoring it to the human body model, as shown in the following figure. Figure 5 As shown. The obtained human body is then randomly rotated in 3D space with a small amplitude to improve the adaptability of the proposed chest reconstruction model to changes in human body posture and reduce the requirements for the standardization accuracy of perceived information.

[0160] Finally, the area around the human chest is rendered as a depth image (e.g., Figure 6 As shown in a), random occlusion was added to simulate the loss of local information caused by the absorption of infrared beams emitted by the sensor by dark skin, thus obtaining human chest simulation data samples for model training and testing (such as...). Figure 6 (as shown in b).

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image, characterized in that, Includes the following steps: Step 1: Generate shape priors by perturbing the postoperative thoracic cavity point cloud with Gaussian noise; Step 2: Adaptively fuse depth image features and shape prior features using the AFF module; Step 3: 3D mesh prediction; The fused features are input into a graph convolutional neural network, which outputs the vertex coordinates of a reconstructed 3D model of the thoracic cavity. Step 4: Root node displacement estimation; Based on global feature information, the relative displacement of the root node of the thoracic cavity model in the world coordinate system is predicted to achieve model space alignment. Step 5: Iterative optimization; The initial prediction results are introduced into the noise and used as a new shape prior. Multiple rounds of prediction and fusion are performed to gradually improve the reconstruction accuracy.

2. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 1, characterized in that, Step 1 includes the following specific steps: A local full-size 3D model of the thoracic cavity is generated based on the obtained depth image. This process uses the function f. LCL This is expressed as formula (1); x=f LCL (I D ) (1) Among them, I D For a depth image, x = (V, E) represents a local full-size chest model, which is composed of N... v The set of vertices V = {v1, v2, ..., v...} Nv } and connecting vertex N e The set of edges of a triangle The resulting 3D topological graphics; Based on formula (1), a shape prior x is given, which contains a rough human body shape. Then formula (1) becomes: x=f LCL (x,I D ) (2) The shape priors x with different confidence levels are generated by adding noise, and the specific process is shown in formula (3); Where ε is Gaussian noise; N(0,1) is a standard normal distribution; t∈{1,2,…,T} is the fuzziness intensity of the current shape prior, which is randomly selected during training; T is set to 1000; α t =1-β t ,β t This represents the fuzzy intensity coefficient at step t, which increases with t, and its value ranges from [5 × 10^t]. -5 5×10 -4 [Achieved linearly;] Further, formula (2) is visualized as formula (4): This expression can be interpreted as using depth image I D The observed shape f D (I D For a rough human body shape f from the shape prior x T (x) is corrected to obtain a chest cavity model that is closer to the real human body; in the above formula Indicates the correction method.

3. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 2, characterized in that, Step 2 includes the following steps: Considering that the credibility of features from different information sources varies for different parts, an AFF module is proposed to implement the correction method in formula (4) through feature fusion. First, the similarity factor S between the shape prior and the depth image information is obtained by calculating the inner product. R The difference between shape prior and true observation is determined, and then it is input into the modified weight mapping layer to obtain the modified weight. Finally, the adaptive fusion of the two features is achieved using formula (5). In the formula, F x F represents the shape characteristics of the fused thoracic cavity. x and F D These are shape features extracted from shape priors and depth images, respectively, both of which are N-type features describing the shape of different parts of the thoracic cavity. x A set of N eigenvectors x It is set to 52; w is a set of weight factors located in the interval [0,1], and the superscript i indicates the i-th element in the set; The Projetion Layer has a weight w p and bias b p An optimizable mapping layer is represented by Equation (6); Among them, S′ R This represents the optimized similarity factor; To ensure that w always lies within the interval [0,1], formulas (7) and (8) are used to... Perform normalization processing; in, Make S′ R To make it positive, max(·) means to take the maximum value from the set.

4. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 2, characterized in that, In step 3, a graph convolution-based Thoracic Decoder neural network is used to convert F... x The result is transformed into a set of local coordinate positions of the vertices, and thus formula (4) is further expressed as formula (9); x=f TD (f AFF (f T (x),f D (I D ))) (9) Among them, f AFF Indicates the AFF module, f TD This indicates Thoracic Decoder.

5. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 2, characterized in that, Step 4 includes the following specific steps: To obtain the position r of the thoracic root node in the world coordinate system g The coarse offset of the thoracic cavity in the basic world coordinate system is extracted from the depth image, and then further corrected using the shape information of the thoracic cavity, thereby obtaining a more accurate world coordinate system position. This process can be represented by formula (10). r g =f r (f Dr (F D )+f xr (F x )) (10) Among them, f Dr Described as a feature-based global average pooling layer, f r For the Root Estimator neural network, and f xr Calculation is based on global average only; The LCL neural network model proposed based on the above concepts includes the following training loss function terms; Local consistency loss is used to describe the difference between the predicted thoracic cavity 3D model x and its true value x, and is specifically expressed as formula (11); L LC =sml1(x,x) (11) Where sml1(·) represents the Smooth loss error function; The deep feature consistency loss is shown in Equation (12). Where mse(·) represents the Mean Squared Error function; The dynamic weighting coefficients are used to describe the fused thoracic cavity shape features F. x The higher the reliability of F, the greater the reliability. x The more credible; where α c The constant weighting factor is set to 10; δ l =L LC For dynamic weighting factors; The shape feature consistency loss is shown in Equation (13). in, Represents the dynamic time weighting coefficient; ζ t As a constant weighting factor, the smaller t decreases, the closer the true shape value is to the prior shape value; Root node consistency loss, used to describe the predicted root node position. Its truth value r g The differences between them are shown in formula (14); Global consistency loss is used to describe the position x of the thoracic cavity model in the predicted world coordinate system. g x between its truth value g The differences are shown in formula (15); L GC = sml1(x g ,x g ) (15) Where x g and x g Obtained through formulas (16) and (17); x g =x+r g (17) Loss of detail consistency, obtained by x g and x g The differences between the rendered depth images further constrain the shape details of the prediction model, and this process is expressed as formula (18); L VGG = vgg(f r (x g (f) r (x g )) (18) Among them, f r (·) represents the renderer function, and vgg(·) represents the VGG loss function; Finally, the above loss terms are integrated according to formula (19) to obtain the integrated loss function for model training; L=ω f ×(ω g ×L GC +oh v ×L VGG )+ω r ×L RC +oh l ×L LC +L DFC +L SFC (19) Where, ω f ω g ω v ω r and ω l These are the weight hyperparameters, set to 10, 0.7, 0.3, 5, and 7 respectively.

6. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 2, characterized in that, In step 5, considering the difference between the initial shape prior and the actual human body shape, an iterative strategy is adopted; this strategy adds noise to the prediction result each time, and re-includes it as a new shape prior in the next round of iteration. As the number of iterations increases, the noise added gradually decreases, and the shape prior gradually approaches the actual shape of the human body.

7. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 6, characterized in that, A simulation dataset was constructed to train the model, and a Zero-shot DomainAdaptation module was embedded during the iteration process to correct the gap between the prediction results and the observations.

8. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 7, characterized in that, The Zero-shot Domain Adaptation module first finds the vertex v in the world coordinate system. g Point p in the recent perceptual point cloud v The distance d between the two is then used in two ways to make v g Close to p v ; The first method is Forward Fitting, as shown in formulas (20) and (21); δ=p v -v g (20) v' g =δ z +v g (21) Where δ is v g Point to p v The vector, δ z This is the 3D offset vector after setting the horizontal and vertical offsets of δ to zero; The second method is Normal Smoothing, as shown in formulas (22) and (23); in, d represents the mean of d, tanh(·) represents the hyperbolic tangent function, and · represents the inner product operation. This indicates that the point cloud surface is at point P. v The normal vector at Δ n This indicates the correction step size, which adaptively adjusts the correction magnitude based on the Euclidean distance between each point and its target position. The first method acts on the anterior surface of the thoracic cavity, and the second method acts on both sides of the thoracic cavity.

9. The method for reconstructing a full-size thoracic cavity 3D model based on a single-view depth image according to claim 7, characterized in that, The process for constructing the simulation dataset is as follows: Using the SMPL human body model as the base template, a full-size model reconstruction simulation dataset with consistent 3D shape style annotations was established. By changing the shape parameters of the SMPL model, human bodies in different postures were generated, and the human body model was modified through the following steps: First, 659 vertices located on the chest were manually selected to form a chest model, which was then placed in a custom coordinate system. The coordinate system had O as the origin in formula (24), the positive x-axis direction was from left to right, the positive y-axis direction was from the feet to the head, and the positive z-axis direction was the front of the body. in These represent the mean values ​​of all vertices of the chest model in the x, y, and z directions, respectively. Then, the area located on the negative x-axis is considered as the left breast, and the rest is the right breast; after the left breast is mirrored and flipped along the yoz plane to the positive x-axis, the left and right breasts are processed separately using formula (25); Where, δ d The expansion factor is calculated as shown in formula (26); and Factors are used to define the regions of shape change in the x and y directions. As shown in formula (27), It has a similar form; d d =s d ×k d ×v i (26) Where, σ d For the expansion symbol, when its vertex v i When located on the affected side, σ d =-1, when located on the healthy side, σ d =1; k d ~[0.01,0.03] is the expansion size factor, and its value is the same for the same breast on the same side of the same human body; Where, x i For vertex v i x-coordinate value, x min and x max These are the minimum and maximum x-coordinates of all vertices on one side of the chest, respectively, when v i When located on the affected side, x min =0; when v i When located on the healthy side, x min =0.7×x max ; Then, the chest model is transformed back to the original coordinate system using the center point O, thus restoring it to the human body model. The resulting human body is then rotated slightly randomly in 3D space. Finally, the area around the human chest is rendered as a depth image, and random occlusion is added to simulate the loss of local information caused by the absorption of infrared beams emitted by the sensor by dark skin, thus obtaining human chest simulation data samples for model training and testing.

Images