Skin surface three-dimensional reconstruction method and system based on 2D Gaussian under respiratory movement

Abstract

The invention provides a 2D Gaussian-based skin surface three-dimensional reconstruction method and system under respiratory movement, and relates to the field of three-dimensional reconstruction, and the method comprises the steps: generating a target point cloud, and obtaining first data information; collecting first feature data information of respiratory movement corresponding to the skin; performing mapping processing to obtain second feature data information; superposing the first data information and the second feature data information, determining target data information, and obtaining a first overall representation model; mapping the first overall representation model to a camera coordinate system to obtain a second overall representation model; and performing rasterization rendering on the second overall representation model to obtain a target image. In the three-dimensional reconstruction process, the influence of respiratory movement on skin surface movement displacement is considered, and the problems that in current skin surface three-dimensional reconstruction, due to the influence of respiratory movement factors, large noise and measurement errors exist, and an image generated through skin surface three-dimensional reconstruction is not ideal are solved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a method and system for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion. Background Art

[0002] 2D Gaussian sputtering is a new method for modeling and reconstructing geometrically accurate radiation fields from multi-view images. It collapses the 3D volume into a set of 2D oriented planar Gaussian ellipses. Unlike 3D Gaussian, 2D Gaussian ellipses provide visual precision while essentially modeling the surface. Figure 1 The perspective-accurate 2D sputtering process using light sputtering intersection and rasterization can accurately restore thin surfaces and achieve stable optimization. The fusion of robotics and ultrasound scanning can overcome the shortcomings of large variability and low repeatability of ultrasound images in traditional ultrasound examinations. However, there are huge differences in the surface morphology of different human bodies, and currently it still relies heavily on manual generation of suitable scanning trajectories for free-form surfaces.

[0003] In the related art, a depth camera can be used to photograph the skin surface, and a point cloud can be generated according to the depth of the depth camera and it is assumed that these point clouds just cover the skin surface. The normals of these point clouds on the skin surface are extracted to plan the robot movement for scanning.

[0004] However, the point cloud generated by the depth camera may be sparse in some areas. For example, in areas with low reflectivity or occlusion, sparse point clouds will cause holes and discontinuities in surface reconstruction, requiring additional interpolation and completion techniques to solve these problems. In addition, the depth camera is easily affected by factors such as ambient light and surface reflection when capturing depth information. The superimposed fluctuations and displacements of the skin surface caused by respiratory movement will lead to large noise and measurement errors, and the image generated by the three-dimensional reconstruction of the skin surface is not ideal. Summary of the invention

[0005] In response to the shortcomings of the prior art, the present application provides a method and system for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion, which solves the problem that the current three-dimensional reconstruction of the skin surface is affected by respiratory motion factors, resulting in large noise and measurement errors, and the image generated by the three-dimensional reconstruction of the skin surface is not ideal.

[0006] To achieve the above objectives, this application is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion, the method comprising:

[0008] Capture skin surface images through a depth camera and generate a target point cloud;

[0009] Using a Gaussian ellipse that obeys a Gaussian distribution to represent skin surface information represented by a target point cloud, obtaining first data information in an ellipse local coordinate system;

[0010] Collecting first characteristic data information of respiratory motion corresponding to the skin through a preset target device, the first characteristic data information including maximum displacement amplitude and respiratory frequency;

[0011] Mapping the first characteristic data information to the local coordinate system of the ellipse to obtain second characteristic data information; wherein the second characteristic data information is used to characterize the influence of the respiratory movement on the position of the ellipse;

[0012] The first data information and the second characteristic data information are superimposed to determine the target data information in the local coordinate system of the ellipse, and obtain a first overall representation model corresponding to the ellipse affected by the respiratory motion;

[0013] Mapping the first overall representation model to the camera coordinate system to obtain a second overall representation model;

[0014] Based on a preset low-pass filtering method, the second overall representation model is rasterized and rendered to perform three-dimensional reconstruction to obtain a target image of the skin surface under respiratory motion.

[0015] According to the first aspect of the embodiment of the present application, the mapping of the first feature data information to the ellipse local coordinate system to obtain the second feature data information may specifically include the following steps:

[0016] The first characteristic data information is mapped to the global coordinate system through a preset first conversion matrix to obtain intermediate data information; wherein the first conversion matrix is ​​a conversion matrix from the skin to the global coordinate system;

[0017] Based on a preset second conversion matrix, the intermediate data information is mapped from the global coordinate system to the ellipse local coordinate system to obtain second feature data information.

[0018] According to the first aspect of the embodiment of the present application, the mapping of the first overall representation model to the camera coordinate system to obtain the second overall representation model may specifically include the following steps:

[0019] Mapping the first overall representation model from the ellipse local coordinate system to the global coordinate system through a second transformation matrix to obtain an intermediate model;

[0020] Based on a preset third transformation matrix, the intermediate model is mapped from the global coordinate system to the camera coordinate system to obtain a second overall representation model.

[0021] According to a first aspect of an embodiment of the present application, the aforementioned target device includes a medical imaging device, a motion capture system, and a respiratory monitoring device;

[0022] The aforementioned collecting of the first characteristic data information of the respiratory movement corresponding to the skin through the preset target device may specifically include the following steps:

[0023] Measuring the maximum displacement of the skin surface during the respiratory cycle by at least one of a medical imaging device and a motion capture system to obtain a maximum displacement amplitude;

[0024] The respiratory movement is monitored by a respiratory monitoring device to obtain the respiratory rate.

[0025] According to the first aspect of the embodiment of the present application, before the first characteristic data information of the respiratory motion corresponding to the skin is collected by the preset target device, the 2D Gaussian-based three-dimensional reconstruction method of the skin surface under the respiratory motion may also include:

[0026] It is assumed that the skin surface displacement caused by respiratory motion changes periodically in a sinusoidal function, the displacement amplitude and period are known or can be measured, and the respiratory frequency is relatively stable, so as to model the respiratory motion;

[0027] Match the center point of the Gaussian ellipse to the position variation of the sinusoidal function on the abdominal skin surface;

[0028] Among them, the skin surface position change caused by respiratory movement satisfies the expression:

[0029] y(t)=Asin(ωt+φ)

[0030] Where y(t) is the displacement of the skin surface at time t, A is the maximum amplitude of the displacement, ω is the angular frequency, ω=2πf, f is the breathing frequency, φ is the initial phase and is used to determine the position of the skin surface at t=0.

[0031] According to the first aspect of the embodiment of the present application, the amplitude of the respiratory motion is limited, and the first tangent vector t corresponding to the Gaussian ellipse is u , the second tangent vector t v and a scaling vector S = (s u ,s v ) remain unchanged;

[0032] The second characteristic data information satisfies the expression;

[0033] Δp k =H T (y(t)w d )

[0034] In the formula, Δp k is the second characteristic data information, W d is the first transformation matrix, H T is the second conversion matrix, and H is the transposed matrix of the second conversion matrix;

[0035] The first overall representation model satisfies the expression:

[0036] ΔP(u)=(p k +Δp k )+s u t u u+s v t v v

[0037] =ΔH·(u+Δp k (u), v+Δp k (v), 1, 1) T

[0038] Function value of the center point of the Gaussian ellipse affected by respiratory motion Satisfies the expression:

[0039]

[0040] in, pk is the first data information, the coordinates of the center point of the Gaussian ellipse in the local ellipse coordinate system are (u, v), Δp k (u) is the tangent vector t u The displacement in the direction, Δp k (v) is the tangent vector t v The displacement in the direction; S s is a diagonal matrix with the last column being 0 and used to characterize the scaling parameters of the Gaussian ellipse; R is the directional parameter of the Gaussian ellipse and satisfies: R = [t u , t v , t w ], t w =t u ×t v .

[0041] According to a first aspect of the embodiment of the present application, the color Δc(x) of a pixel in the target image satisfies the expression:

[0042]

[0043] Among them, c i is the color of the i-th Gaussian ellipse, α i is the transparency of the i-th Gaussian ellipse, because during the change, the color and transparency properties of the Gaussian ellipse do not change; is the normalized Gaussian value of the i-th Gaussian ellipse at Δu(x) after respiratory motion changes, It is the cumulative product of the opacity of the intersection points from the 1st to the i-1th Gaussian ellipse after the respiratory movement changes; Δu(x) satisfies the expression:

[0044]

[0045] In the formula, and They represent the i-th parameter of the homogeneous coordinate plane, 1≤i≤4.

[0046] In the second aspect, an embodiment of the present application provides a 2D Gaussian-based three-dimensional reconstruction system for the skin surface under respiratory motion, the system comprising a first acquisition module, a representation module, a second acquisition module, a first mapping module, an overlay module, a second mapping module and a rendering module; wherein the first acquisition module is used to acquire an image of the skin surface through a depth camera and generate a target point cloud; the representation module is used to use a Gaussian ellipse that obeys a Gaussian distribution to represent the skin surface information represented by the target point cloud, and obtain first data information in the local coordinate system of the ellipse; the second acquisition module is used to acquire first characteristic data information of the respiratory motion corresponding to the skin through a preset target device, and the first characteristic data information includes a maximum displacement amplitude and a respiratory frequency rate; the first mapping module is used to map the first feature data information to the local coordinate system of the ellipse to obtain the second feature data information; wherein the second feature data information is used to characterize the influence of respiratory motion on the position of the ellipse; the superposition module is used to superimpose the first data information and the second feature data information, determine the target data information in the local coordinate system of the ellipse, and obtain the first overall representation model corresponding to the ellipse affected by the respiratory motion; the second mapping module is used to map the first overall representation model to the camera coordinate system to obtain the second overall representation model; the rendering module is used to rasterize the second overall representation model based on a preset low-pass filtering method, so as to perform three-dimensional reconstruction to obtain the target image of the skin surface under respiratory motion.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the 2D Gaussian-based three-dimensional reconstruction method of the skin surface under respiratory motion in the first aspect described above is implemented.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion in the first aspect mentioned above is implemented.

[0049] This application provides a method and system for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion. Compared with the prior art, it has the following beneficial effects:

[0050] In the process of three-dimensional reconstruction, the present application takes into account the influence of respiratory movement on the movement and displacement of the skin surface. On the one hand, the present application collects skin surface images through a depth camera to generate a target point cloud, and processes it based on 2D Gaussian technology to obtain first data information; on the other hand, the first characteristic data information of the respiratory movement is collected through the target device to determine the maximum displacement amplitude and respiratory frequency corresponding to the respiratory movement, and the first characteristic data information is mapped to the elliptical local coordinate system to obtain the second characteristic data information to be consistent with the aforementioned first data information, and then the data is superimposed to obtain the target data information, and then the first overall representation model that characterizes the real information of the skin under the influence of respiratory movement is obtained. For image rendering, the first overall representation model is mapped to the camera coordinate system to obtain the second overall representation model. In order to avoid the Gaussian ellipse from degenerating into a straight line during the rendering process, the present application performs rasterization rendering based on a low-pass filtering method to obtain an accurate target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 It is a flow chart of a method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion provided in an embodiment of the present application;

[0053] Figure 2 It is a structural schematic diagram of a 2D Gaussian-based three-dimensional reconstruction system of skin surface under respiratory motion provided in an embodiment of the present application;

[0054] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0057] The embodiments of the present application provide a method and system for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion, thereby solving the problem that the current three-dimensional reconstruction of the skin surface is affected by respiratory motion factors, resulting in large noise and measurement errors, and the image generated by the three-dimensional reconstruction of the skin surface is not ideal.

[0058] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0059] In the field of radiation field reconstruction, 3D Gaussian sputtering can achieve high-quality new view synthesis and fast rendering speed. However, due to the multi-view inconsistency of 3D Gaussian, 3D Gaussian sputtering cannot accurately represent the surface. 2D Gaussian sputtering is a new method for modeling and reconstructing geometrically accurate radiation fields from multi-view images, which folds the 3D volume into a set of 2D oriented planar Gaussian ellipses.

[0060] Unlike a 3D Gaussian, a 2D Gaussian ellipse essentially models the surface while providing visual Figure 1 The perspective-accurate 2D sputtering process using ray sputtering intersection and rasterization can accurately restore thin surfaces and achieve stable optimization. In addition, the depth distortion and normal consistency terms are used to further improve the quality of reconstruction; the differentiable renderer proposed by this method can achieve noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed and real-time rendering.

[0061] In related technologies, the integration of robotics and ultrasound scanning can overcome the shortcomings of large variability and low repeatability of ultrasound images in traditional ultrasound examinations. However, there are huge differences in the surface morphology of different human bodies, and currently it still relies heavily on manually generating suitable scanning trajectories for free-form surfaces. A depth camera can be used to capture the skin surface, and a point cloud can be generated based on the depth of the depth camera and it is assumed that these point clouds just cover the skin surface. The normals of these point clouds on the skin surface can be extracted to plan the robot movement for scanning.

[0062] However, the point cloud generated by the depth camera may be sparse in some areas, such as in areas with low reflectivity or occlusion. Sparse point clouds can lead to holes and discontinuities in surface reconstruction, requiring additional interpolation and completion techniques to solve these problems. In addition, the depth camera is easily affected by factors such as ambient light and surface reflection when capturing depth information, resulting in noise and measurement errors. In addition, processing a large amount of point cloud data requires high computing resources and time, making it difficult to meet the needs of real-time applications.

[0063] Furthermore, affected by respiratory movement, there are fluctuating displacements on the skin surface, which will further increase noise and measurement errors, making the image generated by three-dimensional reconstruction of the skin surface less than ideal and distorted.

[0064] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0065] The following first introduces a 2D Gaussian-based three-dimensional reconstruction method of the skin surface under respiratory motion provided by an embodiment of the present application.

[0066] The present application provides a flowchart of a method for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion, as shown in FIG. Figure 1 As shown, the method may include the following steps S110-S170.

[0067] S110, collecting skin surface images through a depth camera and generating a target point cloud.

[0068] S120 , using a Gaussian ellipse that obeys a Gaussian distribution to represent skin surface information represented by a target point cloud, and obtaining first data information in an ellipse local coordinate system.

[0069] S130. Collect first characteristic data information corresponding to the respiratory movement of the skin through a preset target device, where the first characteristic data information includes a maximum displacement amplitude and a respiratory frequency.

[0070] S140, mapping the first characteristic data information to the local coordinate system of the ellipse to obtain second characteristic data information; wherein the second characteristic data information is used to characterize the influence of the respiratory movement on the position of the ellipse.

[0071] S150, superimposing the first data information and the second characteristic data information, determining the target data information in the local coordinate system of the ellipse, and obtaining a first overall representation model corresponding to the ellipse affected by the respiratory motion.

[0072] S160 , mapping the first overall representation model to a camera coordinate system to obtain a second overall representation model.

[0073] S170 , based on a preset low-pass filtering method, rasterizing and rendering the second overall representation model to perform three-dimensional reconstruction to obtain a target image of the skin surface under respiratory motion.

[0074] The above is a specific implementation method of the three-dimensional reconstruction method of the skin surface based on 2D Gaussian under respiratory motion provided in an embodiment of the present application. In the process of three-dimensional reconstruction, the present application takes into account the influence of respiratory motion on the movement displacement of the skin surface. On the one hand, the present application collects skin surface images through a depth camera to generate a target point cloud, and processes it based on 2D Gaussian technology to obtain the first data information.

[0075] On the other hand, in order to determine the impact of respiratory movement on the skin surface, the present application collects the first characteristic data information of respiratory movement through a target device to determine the maximum displacement amplitude and respiratory frequency corresponding to the respiratory movement, and maps the first characteristic data information to an elliptical local coordinate system to obtain second characteristic data information to be consistent with the aforementioned first data information, and then performs data superposition to obtain target data information, and then obtains a first overall representation model that characterizes the real information of the skin under the influence of respiratory movement.

[0076] It should also be noted that image rendering needs to be performed in the camera coordinate system. Since the aforementioned first overall representation model is in the elliptical local coordinate system, in order to perform image rendering, the first overall representation model is mapped to the camera coordinate system to obtain a second overall representation model; in addition, in order to avoid the Gaussian ellipse from degenerating into a straight line during the rendering process, the present application performs rasterization rendering based on a low-pass filtering method, which can obtain an accurate target image through three-dimensional reconstruction.

[0077] This application aims to propose a new method to complete the task of skin surface reconstruction under respiratory movement. First, 2D Gaussian sputtering can naturally generate a smooth surface by using a Gaussian ellipse that satisfies the Gaussian distribution to represent the surface; the smoothness of the Gaussian function enables this method to effectively avoid the discontinuity and mutation of the generated point cloud caused by inaccurate depth detection of the depth camera during the reconstruction process. 2D Gaussian sputtering can handle various complex surface geometries, does not depend on the density of the point cloud, and can generate reasonable surfaces even when the data is sparse; when processing incomplete or partially missing depth data, it can still maintain a good reconstruction effect. The calculation process of 2D Gaussian sputtering can be parallelized and can be accelerated on the GPU using parallel computing technologies such as CUDA, which greatly improves the efficiency of processing large-scale data and makes real-time processing and rendering possible.

[0078] In some embodiments, the first feature data information is mapped to the ellipse local coordinate system to obtain the second feature data information, that is, the aforementioned S140 may specifically include the following steps:

[0079] S210. Map the first characteristic data information to the global coordinate system through a preset first conversion matrix to obtain intermediate data information; wherein the first conversion matrix is ​​a conversion matrix from the skin to the global coordinate system.

[0080] S220 . Based on a preset second conversion matrix, map the intermediate data information from the global coordinate system to the ellipse local coordinate system to obtain second feature data information.

[0081] In the embodiment of the present application, it can be understood that, since the first data information is in the elliptical local coordinate system, in order to weight the influence of respiratory motion on the first data information, it is necessary to perform mapping processing between multiple coordinate systems for the first data information. Specifically, the present application first maps the first data information to the global coordinate system, and processes it based on the second conversion matrix based on the standard between the global coordinate system and the local elliptical coordinate system to obtain the second feature data information in the elliptical local coordinate system for superposition processing with the first data information.

[0082] In some embodiments, the mapping of the first overall representation model to the camera coordinate system to obtain the second overall representation model, that is, the aforementioned S160 may specifically include the following steps:

[0083] S310 . Map the first overall representation model from the ellipse local coordinate system to the global coordinate system through a second transformation matrix to obtain an intermediate model.

[0084] S320. Based on a preset third transformation matrix, map the intermediate model from the global coordinate system to the camera coordinate system to obtain a second overall representation model.

[0085] In the embodiment of the present application, it can be understood that after superimposing the first data information and the second feature data information and obtaining the first overall representation model corresponding to the ellipse affected by the respiratory movement, since the rasterized rendering of the image needs to be performed in the camera coordinate system, the first overall representation model still corresponds to the local coordinate system of the ellipse.

[0086] Based on this, the present application first maps the first overall representation matrix to a standard global coordinate system through the aforementioned second transformation matrix, and then obtains a second overall representation model in the camera coordinate system based on the third transformation matrix. It can be understood that the third transformation matrix is ​​a transformation matrix between a preset global coordinate system and a camera coordinate system.

[0087] In some embodiments, the target device includes a medical imaging device, a motion capture system, and a respiratory monitoring device. The aforementioned collecting the first characteristic data information of the respiratory motion corresponding to the skin through the preset target device, that is, the aforementioned S130 may specifically include the following steps:

[0088] S410, measuring the maximum displacement of the skin surface during the respiratory cycle by at least one of a medical imaging device and a motion capture system to obtain a maximum displacement amplitude.

[0089] S420. Monitor the respiratory movement by using a respiratory monitoring device to obtain the respiratory frequency.

[0090] In the embodiments of the present application, it can be understood that respiratory movement will cause a small displacement of the skin surface. By measuring information such as the maximum displacement amplitude and the respiratory frequency, it is helpful to quantitatively determine the degree of influence of the respiratory movement on the skin surface state, and then facilitate the restoration of the real skin surface information through three-dimensional reconstruction.

[0091] In some embodiments, before the first characteristic data information of the respiratory motion corresponding to the skin is collected by the preset target device, that is, before the aforementioned S130, the 2D Gaussian-based three-dimensional reconstruction method of the skin surface under respiratory motion may specifically include the following steps:

[0092] S510. Assume that the skin surface displacement caused by respiratory motion varies periodically like a sine function, the displacement amplitude and period are known or measurable, and the respiratory frequency is relatively stable, so as to model the respiratory motion.

[0093] S520, matching the center point of the Gaussian ellipse with the position change of the sinusoidal function on the abdominal skin surface.

[0094] Among them, the skin surface position change caused by respiratory movement satisfies the expression:

[0095] y(t)=Asin(ωt+φ)

[0096] Where y(t) is the displacement of the skin surface at time t, A is the maximum amplitude of the displacement, ω is the angular frequency, ω=2πf, f is the breathing frequency, φ is the initial phase and is used to determine the position of the skin surface at t=0.

[0097] In the embodiments of the present application, it can be understood that respiratory movement is one of the natural physiological activities of the human body, which will produce periodic changes in the position of the skin surface. In order to accurately model this transformation, the present application proposes a method for modeling the skin surface position using a sine function; by modeling the respiratory movement, the surface position change caused by the respiratory movement is modeled as a sine function. After the surface is reconstructed by 2D Gaussian sputtering, the expression of the surface becomes a two-dimensional Gaussian ellipse covering the skin surface. It is only necessary to process these Gaussian ellipses so that their center points match the position changes of the sine function on the abdominal surface. Thanks to the real-time reconstruction effect of 2D Gaussian sputtering, real-time reconstruction of the abdominal surface can be achieved.

[0098] In one example, due to the limited amplitude of respiratory motion, the first tangent vector t corresponding to the Gaussian ellipse isu , the second tangent vector t v and a scaling vector S = (s u ,s v ) remain unchanged;

[0099] The second characteristic data information satisfies the expression;

[0100] Δp k =H T (y(t)w d )

[0101] In the formula, Δp k is the second characteristic data information, W d is the first transformation matrix, H T is the second conversion matrix, and H is the transposed matrix of the second conversion matrix;

[0102] The first overall representation model satisfies the expression:

[0103] ΔP(u)=(p k +Δp k )+s u t u u+s v t v v

[0104] =ΔH·(u+Δp k (u), v+Δp k (v), 1, 1) T

[0105] Function value of the center point of the Gaussian ellipse affected by respiratory motion Satisfies the expression:

[0106]

[0107] in, p k is the first data information, the coordinates of the center point of the Gaussian ellipse in the local ellipse coordinate system are (u, v), Δp k (u) is the tangent vector t u The displacement in the direction, Δp k (v) is the tangent vector t v The displacement in the direction; S s is a diagonal matrix with the last column being 0 and used to characterize the scaling parameters of the Gaussian ellipse; R is the directional parameter of the Gaussian ellipse and satisfies: R = [t u , t v , t w ], t w =t u ×t v .

[0108] In the embodiment of the present application, the Gaussian basis element in the 2D Gaussian sputtering is a Gaussian ellipse, and the major and minor axes of the Gaussian ellipse satisfy the Gaussian distribution. u and the tangent vector t v It can be understood as the unit vector in the direction of the major and minor axes of the Gaussian ellipse. The change of the tangent vector indicates that the Gaussian ellipse has rotated. After multiplying it by the corresponding scaling vector, it is the length of the major and minor axes of the Gaussian ellipse. In addition, the normal of the Gaussian ellipse perpendicular to the elliptical surface can be calculated by the cross product of the tangent vector t w =t u ×t v Represents; then the directional parameter of a 2D Gaussian ellipse can be expressed as R = [t u , t v , t w ]; Since the normal of the Gaussian ellipse is obtained according to the tangent vector, and the normal is a unit vector and does not affect the shape of the Gaussian ellipse, the scalability parameters of the 2D Gaussian ellipse can be expressed as a diagonal matrix S with the last column being 0 s .

[0109] In one example, Δp k (u) and Δp k (v) satisfies the expression:

[0110] h u ·(u+Δp k (u), v+Δp k (v), 1, 1) T

[0111] =h v ·(u+Δp k (u), v+Δp k (v), 1, 1) T =0

[0112] The above formula can be derived:

[0113] Δu(x) satisfies the expression:

[0114]

[0115] Δv(x) satisfies the expression:

[0116]

[0117] In the formula, and They represent the i-th parameter of the homogeneous coordinate plane, 1≤i≤4.

[0118] A common strategy for rendering a 2D Gaussian ellipse is to use the affine approximation of perspective projection to project the 2D Gaussian ellipse onto the image plane; the 2D-to-2D mapping used is the mapping of the 2D ellipse in space to the 2D plane. Since 2D-to-2D will cause the Gaussian ellipse to degenerate into a straight line, rasterization rendering will not be possible. In order to alleviate this problem, this application uses a low-pass filter. To mitigate this situation:

[0119]

[0120] Where Δx c is the center point of the 2D Gaussian ellipse after respiratory motion changes projected onto the image plane; σ represents the radius after projection.

[0121] In one example, the present application calculates the final color of a pixel according to a hybrid method, and the color Δc(x) of the pixel in the target image satisfies the expression:

[0122]

[0123] Among them, c i is the color of the i-th Gaussian ellipse, α i is the transparency of the i-th Gaussian ellipse, because during the change, the color and transparency properties of the Gaussian ellipse do not change; is the normalized Gaussian value of the i-th Gaussian ellipse at Δu(x) after respiratory motion changes, It is the cumulative product of the opacity of the intersection points from the 1st to the i-1th Gaussian ellipses after the respiratory movement changes.

[0124] In some embodiments, Figure 2 As shown, the present application also provides a 2D Gaussian-based three-dimensional skin surface reconstruction system 600 under respiratory motion, and the three-dimensional skin surface reconstruction system 600 may include the following modules:

[0125] The first acquisition module 610 is used to acquire an image of the skin surface through a depth camera and generate a target point cloud.

[0126] The representation module 620 is used to use a Gaussian ellipse that obeys a Gaussian distribution to represent the skin surface information represented by the target point cloud, and obtain first data information in the ellipse local coordinate system.

[0127] The second acquisition module 630 is used to acquire first characteristic data information of the respiratory movement corresponding to the skin through a preset target device, where the first characteristic data information includes a maximum displacement amplitude and a respiratory frequency.

[0128] The first mapping module 640 is used to map the first characteristic data information to the local coordinate system of the ellipse to obtain the second characteristic data information; wherein the second characteristic data information is used to characterize the influence of the respiratory movement on the position of the ellipse.

[0129] The superposition module 650 is used to superimpose the first data information and the second characteristic data information, determine the target data information in the local coordinate system of the ellipse, and obtain a first overall representation model corresponding to the ellipse affected by the respiratory motion.

[0130] The second mapping module 660 is used to map the first overall representation model to the camera coordinate system to obtain a second overall representation model.

[0131] The rendering module 670 is used to perform raster rendering on the second overall representation model based on a preset low-pass filtering method to perform three-dimensional reconstruction to obtain a target image of the skin surface under respiratory motion.

[0132] According to an embodiment of the present application, any multiple modules among the first acquisition module 610, the representation module 620, the second acquisition module 630, the first mapping module 640, the overlay module 650, the second mapping module 660 and the rendering module 670 may be combined into one module for implementation, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module.

[0133] In some embodiments, the first mapping module 640 may be specifically used to:

[0134] The first characteristic data information is mapped to the global coordinate system through a preset first conversion matrix to obtain intermediate data information; wherein the first conversion matrix is ​​a conversion matrix from the skin to the global coordinate system;

[0135] Based on a preset second conversion matrix, the intermediate data information is mapped from the global coordinate system to the ellipse local coordinate system to obtain second feature data information.

[0136] In some embodiments, the second mapping module 660 may be specifically used to:

[0137] The first overall representation model is mapped from the ellipse local coordinate system to the global coordinate system through the second transformation matrix to obtain an intermediate model.

[0138] Based on a preset third transformation matrix, the intermediate model is mapped from the global coordinate system to the camera coordinate system to obtain a second overall representation model.

[0139] In some embodiments, the second acquisition module 630 may be specifically used for:

[0140] The maximum displacement of the skin surface during the respiratory cycle is measured by at least one of a medical imaging device and a motion capture system to obtain a maximum displacement amplitude.

[0141] The respiratory movement is monitored by a respiratory monitoring device to obtain the respiratory rate.

[0142] In some embodiments, the skin surface 3D reconstruction system 600 may further include a modeling hypothesis module 680, which may be specifically used to:

[0143] It is assumed that the skin surface displacement caused by respiratory motion changes periodically in a sinusoidal function, the displacement amplitude and period are known or can be measured, and the respiratory frequency is relatively stable, so as to model the respiratory motion;

[0144] The center point of the Gaussian ellipse was matched to the sinusoidal function position variation on the abdominal skin surface.

[0145] Figure 2 Each module in the system shown has the function of implementing each step in the aforementioned 2D Gaussian-based three-dimensional reconstruction method of the skin surface under respiratory motion, and can achieve its corresponding technical effects. For the sake of concise description, it will not be repeated here.

[0146] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 3 shown.

[0147] The electronic device may include a processor 710 and a memory 720 storing computer program instructions.

[0148] Specifically, the processor 710 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0149] The memory 720 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 720 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 720 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 720 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 720 is a non-volatile solid-state memory.

[0150] The memory 720 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 720 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the above-mentioned embodiments of the method for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion.

[0151] The processor 710 reads and executes the computer program instructions stored in the memory 720 to implement any one of the 2D Gaussian-based three-dimensional reconstruction methods for the skin surface under respiratory motion in the above embodiments.

[0152] In one example, the electronic device may further include a communication interface 730 and a bus 700. Figure 3 As shown, the processor 710, the memory 720, and the communication interface 730 are connected via a bus 700 and communicate with each other.

[0153] The communication interface 730 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0154] Bus 700 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 700 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0155] In addition, in combination with the 2D Gaussian-based three-dimensional reconstruction method of the skin surface under respiratory motion in the above embodiments, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the 2D Gaussian-based three-dimensional reconstruction methods of the skin surface under respiratory motion in the above embodiments is implemented.

[0156] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0157] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0158] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0159] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0160] In summary, compared with the prior art, this application has the following beneficial effects:

[0161] 1. In the process of three-dimensional reconstruction, the present application takes into account the influence of respiratory movement on the movement and displacement of the skin surface, collects skin surface images through a depth camera to generate a target point cloud, and processes it based on 2D Gaussian technology to obtain first data information; the first feature data information of the respiratory movement is also collected through a target device and mapped to an elliptical local coordinate system to obtain second feature data information to be consistent with the aforementioned first data information, and the target data information is obtained by data superposition, thereby obtaining a first overall representation model that characterizes the real information of the skin under the influence of respiratory movement.

[0162] 2. During the image rendering process, the present application maps the first overall representation model to the camera coordinate system and obtains the second overall representation model. In order to avoid the Gaussian ellipse from degenerating into a straight line during the rendering process, rasterization rendering is performed based on a low-pass filtering method to obtain an accurate target image.

[0163] 3. This application uses an implicit expression method to perform three-dimensional reconstruction of the skin surface, model respiratory movements, match the center point of the Gaussian ellipse with the position change of the sinusoidal function on the skin surface, use parallel computing technologies such as CUDA to accelerate on the GPU, and use implicit methods to model the skin surface. This application is based on the real-time reconstruction effect of 2D Gaussian sputtering, and can achieve real-time reconstruction of the abdominal surface.

[0164] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D reconstruction method of skin surface based on 2D Gaussian under respiratory motion, characterized in that: include: Capture skin surface images through a depth camera and generate a target point cloud; Using a Gaussian ellipse that obeys a Gaussian distribution to represent the skin surface information represented by the target point cloud, to obtain first data information in an ellipse local coordinate system; Collecting first characteristic data information of respiratory movement corresponding to the skin through a preset target device, wherein the first characteristic data information includes a maximum displacement amplitude and a respiratory frequency; Mapping the first characteristic data information to the local coordinate system of the ellipse to obtain second characteristic data information; wherein the second characteristic data information is used to characterize the influence of respiratory motion on the position of the ellipse; Superimposing the first data information and the second characteristic data information, determining the target data information in the local coordinate system of the ellipse, and obtaining a first overall representation model corresponding to the ellipse affected by the respiratory motion; Mapping the first overall representation model to a camera coordinate system to obtain a second overall representation model; Based on a preset low-pass filtering method, the second overall representation model is rasterized and rendered to perform three-dimensional reconstruction to obtain a target image of the skin surface under respiratory movement.

2. The method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion according to claim 1, characterized in that: The mapping the first feature data information to the ellipse local coordinate system to obtain the second feature data information includes: The first characteristic data information is mapped to the global coordinate system through a preset first conversion matrix to obtain intermediate data information; wherein the first conversion matrix is ​​a conversion matrix from the skin to the global coordinate system; Based on a preset second conversion matrix, the intermediate data information is mapped from the global coordinate system to the ellipse local coordinate system to obtain second feature data information.

3. The method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion as claimed in claim 2, characterized in that: Mapping the first overall representation model to a camera coordinate system to obtain a second overall representation model includes: Mapping the first overall representation model from the ellipse local coordinate system to the global coordinate system through the second transformation matrix to obtain an intermediate model; Based on a preset third transformation matrix, the intermediate model is mapped from the global coordinate system to the camera coordinate system to obtain a second overall representation model.

4. The method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion according to claim 1, characterized in that: The target devices include medical imaging equipment, motion capture systems, and respiratory monitoring equipment; The collecting of first characteristic data information of respiratory motion corresponding to the skin through a preset target device includes: Measuring the maximum displacement of the skin surface during the respiratory cycle by at least one of the medical imaging device and the motion capture system to obtain the maximum displacement amplitude; The respiratory movement is monitored by the respiratory monitoring device to obtain the respiratory frequency.

5. The method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion according to claim 1 or 4, characterized in that: Before collecting the first characteristic data information of the respiratory motion corresponding to the skin through the preset target device, the 2D Gaussian-based three-dimensional reconstruction method of the skin surface under the respiratory motion also includes: It is assumed that the skin surface displacement caused by respiratory motion changes periodically in a sinusoidal function, the displacement amplitude and period are known or can be measured, and the respiratory frequency is relatively stable, so as to model the respiratory motion; Matching the center point of the Gaussian ellipse with the position change of the sinusoidal function on the abdominal skin surface; Among them, the skin surface position change caused by respiratory movement satisfies the expression: y(t)=Asin(ωt+φ) Wherein, y(t) is the displacement of the skin surface at time t, A is the maximum amplitude of the displacement, ω is the angular frequency, ω=2πf, f is the breathing frequency, φ is the initial phase and is used to determine the position of the skin surface at t=0.

6. The method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion according to claim 5, characterized in that: The amplitude of the respiratory motion is limited, and the first tangent vector t corresponding to the Gaussian ellipse u , the second tangent vector t v and a scaling vector S = (s u ,s v ) remain unchanged; The second characteristic data information satisfies the expression; Δp k =H T (y(t)w d ) In the formula, Δp k is the second characteristic data information, W d is the first transformation matrix, H T is the second conversion matrix, and H is the transposed matrix of the second conversion matrix; The first overall representation model satisfies the expression: AP(u)=(p k +Δp k )+s u t u u+s v t v v =ΔH·(u+Δp k (u),v+Δp k (v),1,1) T Function value of the center point of the Gaussian ellipse affected by respiratory motion Satisfies the expression: in, pk is the first data information, the coordinates of the center point of the Gaussian ellipse in the local coordinate system of the ellipse are (u, v), Δp k (u) is the tangent vector t u The displacement in the direction, Δp k (v) is the tangent vector t v The displacement in the direction; S S is a diagonal matrix with the last column being 0 and used to characterize the scaling parameters of the Gaussian ellipse; R is the directional parameter of the Gaussian ellipse and satisfies: R = [t u , t v , t w ], t w =t u ×t v .

7. The method for three-dimensional reconstruction of skin surface based on 2D Gaussian under respiratory motion according to claim 1, characterized in that: The color Δc(x) of the pixel in the target image satisfies the expression: Among them, c i is the color of the i-th Gaussian ellipse, α i is the transparency of the i-th Gaussian ellipse, because during the change, the color and transparency properties of the Gaussian ellipse do not change; is the normalized Gaussian value of the i-th Gaussian ellipse at Δu(x) after respiratory motion changes, It is the cumulative product of the opacity of the intersection points from the 1st to the i-1th Gaussian ellipse after the respiratory movement changes; Δu(x) satisfies the expression: In the formula, and They represent the i-th parameter of the homogeneous coordinate plane, 1≤i≤4.

8. A 3D reconstruction system of skin surface based on 2D Gaussian under respiratory motion, characterized in that: include: A first acquisition module, used to acquire an image of the skin surface through a depth camera and generate a target point cloud; A representation module, configured to represent the skin surface information represented by the target point cloud using a Gaussian ellipse that obeys a Gaussian distribution, and obtain first data information in an ellipse local coordinate system; A second acquisition module is used to acquire first characteristic data information of respiratory motion corresponding to the skin through a preset target device, wherein the first characteristic data information includes a maximum displacement amplitude and a respiratory frequency; A first mapping module is used to map the first characteristic data information to the local coordinate system of the ellipse to obtain second characteristic data information; wherein the second characteristic data information is used to characterize the influence of respiratory movement on the position of the ellipse; A superposition module, used for superimposing the first data information and the second characteristic data information, determining the target data information in the local coordinate system of the ellipse, and obtaining a first overall representation model corresponding to the ellipse affected by the respiratory motion; A second mapping module, used for mapping the first overall representation model to a camera coordinate system to obtain a second overall representation model; A rendering module is used to perform raster rendering on the second overall representation model based on a preset low-pass filtering method to perform three-dimensional reconstruction to obtain a target image of the skin surface under respiratory movement.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the method for three-dimensional reconstruction of the skin surface based on 2D Gaussian under respiratory motion as described in any one of claims 1 to 7 is implemented.

Images