Two-Modal Contour Scanning Imaging Method, Device, Equipment and Storage Medium

Abstract

The present invention discloses a two-modal based contour scanning imaging method, device, equipment and storage medium. First, a two-dimensional image and an infrared image of a human body to be measured are obtained. Then, phase unwrapping is performed on the two-dimensional image. Then, special points in the current body surface point cloud are filtered, and based on the coordinate transformation relationship, the current body surface point cloud is projected onto the infrared image. Points in the current body surface point cloud within the target area are extracted for point cloud registration, and the difference between the current body surface point cloud and the reference body surface point cloud is calculated to obtain the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud. Finally, rendering is performed based on the current body surface point cloud to obtain a target image. Based on ordinary three-dimensional imaging, the present invention adds an infrared image, increasing the data dimension from three dimensions to four dimensions, which can weaken the registration sliding effect and improve the speed and accuracy of registration.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scanning, and more specifically, to a two-modal based contour scanning imaging method, device, equipment and storage medium. Background Art

[0002] The three-dimensional information of the body surface refers to the three-dimensional shape, size and position information of the human body or object surface obtained through three-dimensional scanning technology. The acquisition of the three-dimensional information of the body surface mainly relies on three-dimensional scanning technology. Through non-contact measurement methods, three-dimensional scanning technology can quickly and accurately obtain the three-dimensional data of the object surface. Three-dimensional scanning technology is widely used in medical, industrial design, forensic medicine, film and television special effects and other fields. In the medical field, it can be applied to the surface guidance system in radiotherapy to collect the three-dimensional information of the body surface of patients before and during treatment, so as to accurately position the patients before and during radiotherapy, so as to realize precise radiotherapy for the affected part.

[0003] Currently, in order to obtain the three-dimensional information of the human body surface, a scanning imaging system projects a pattern on the human body surface, where the projected pattern includes but is not limited to one-dimensional line structured light and two-dimensional patterns. During the three-dimensional body surface scanning process, the camera collects the pattern projected on the human body surface. If the surface is flat, the pattern captured by the camera is similar to the original pattern projected by the projector; if the surface is uneven, the pattern captured by the camera will be distorted. At this time, it is necessary to calculate the depth information of the target point according to the spatial positions of the camera and the projector.

[0004] The core of the three-dimensional scanning system is to compare the human body surface with the reference surface to detect the spatial positioning deviation. In the field of computer vision, the Iterative Closest Point (ICP) algorithm is a common point cloud registration method. Among them, the ICP algorithm continuously iterates to reduce the distance between two point clouds. However, when the ICP algorithm is used to register objects with translational or rotational symmetry, a registration sliding effect will occur. For example, it is more likely to occur in areas such as the chest and flat abdomen of the human body, resulting in lower speed and accuracy of point cloud registration. Summary of the Invention

[0005] In view of this, the present invention provides a two-modal based contour scanning imaging method, device, equipment and storage medium, which is used to solve the problem that in the process of using the ICP algorithm to register all points in two human body point clouds in the existing three-dimensional scanning imaging method, a registration sliding effect is likely to occur, resulting in lower speed and accuracy of human body point cloud registration.

[0006] To achieve the above object, the following solutions are proposed:

[0007] A two-modal based contour scanning imaging method, the method includes:

[0008] Obtain the two-dimensional image and infrared image of the human body to be measured;

[0009] Perform phase unwrapping on the two-dimensional image to obtain the current body surface point cloud;

[0010] Filter the special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on the preset coordinate transformation relationship, and determine the target area according to the infrared image;

[0011] Extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud;

[0012] Render based on the current body surface point cloud to obtain the target image.

[0013] Preferably, the process of rendering based on the current body surface point cloud to obtain the target image includes:

[0014] Use multi-channel rendering + surface splash + point sprite rendering technology to render the current body surface point cloud.

[0015] Preferably, the process of using multi-channel rendering to render the current body surface point cloud includes:

[0016] Perform multi-channel rendering on the current body surface point cloud through three rendering channels;

[0017] Among them, the processes of rendering by the three rendering channels are as follows:

[0018] The first rendering channel: Enable depth testing, disable color writing, enable depth writing, disable blending, disable point sprites, adjust the depth of each fragment in the particle according to the particle radius, render the current body surface point cloud, and output the depth texture map;

[0019] The second rendering channel: Disable depth testing, enable point sprites, enable color writing, disable depth writing, enable blending, set the blending formula, use the depth values of each fragment in the first rendering channel to adjust the color values of each fragment, render the depth texture map, and output the color texture map;

[0020] The third rendering channel: Disable blending, draw a full-screen quadrilateral in the normalized device coordinate system, and paste the color texture map of the second rendering channel.

[0021] Preferably, the process of performing phase unwrapping on the two-dimensional image by the four-step phase-shifting method and the multi-frequency heterodyne method includes:

[0022] Adopt the four-step phase-shifting method to obtain the phases to be modulated of the two-dimensional image;

[0023] Based on the phase to be modulated, the principal value of the phase is obtained by solving through the arctangent function;

[0024] Calculate by the multi-frequency heterodyne method to obtain a unique principal value within [0, 2π), calculate the phase difference of the sine gratings of different frequencies, convert the high-frequency phase into a low-frequency phase, so that the phase difference signal covers the entire field of view, and then obtain the absolute phase distribution of the two-dimensional image according to the phase difference.

[0025] Preferably, the process of projecting the current body surface point cloud onto the infrared image based on a preset coordinate conversion relationship includes:

[0026] According to the conversion relationship between the point cloud coordinate system and the infrared camera coordinate system, convert the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system to obtain the infrared camera coordinates of each point. Among them, the origin of the infrared camera coordinate system is the optical center of the lens, the x and y axes are parallel to both sides of the imaging plane respectively, and the z axis is the optical axis of the lens;

[0027] According to the conversion relationship between the infrared camera coordinate system and the infrared image coordinate system, convert the infrared camera coordinates of each point from the infrared camera coordinate system to the infrared image coordinate system to obtain the infrared image coordinates of each point. Among them, the origin of the infrared image coordinate system is the midpoint of the imaging plane, and the x and y axes are parallel to both sides of the imaging plane respectively.

[0028] Preferably, the process of converting the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system includes:

[0029] Convert the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system through the formula. The formula is as follows:

[0030] [x c y c z c T =R[x w y w z w T +t,

[0031] where x w , y w , z w are the three coordinate components of point P in the point cloud coordinate system w respectively, x c , y c , z c are the three coordinate components of point P in the infrared camera coordinate system c, and R and t are two transformation matrices;

[0032] The process of converting the infrared camera coordinates of each point from the infrared camera coordinate system to the infrared image coordinate system includes: ​​

[0033] Based on the principle of similar triangles, the infrared camera coordinates of each point are converted from the point cloud coordinate system to the infrared camera coordinate system through a formula, as follows:

[0034]

[0035] Preferably, after extracting the body surface point cloud in the target area for point cloud registration and calculating the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud, it further includes:

[0036] Register the current body surface point cloud with the reference body surface point cloud to obtain a single-modal registration result;

[0037] Perform weighted averaging on the target area point cloud registration result and the single-modal registration result to obtain a target registration result.

[0038] A two-modal based contour scanning imaging device further includes:

[0039] An image acquisition unit for acquiring a two-dimensional image and an infrared image of the human body to be measured;

[0040] A point cloud acquisition unit for performing phase unwrapping on the two-dimensional image to obtain the current body surface point cloud;

[0041] A point cloud processing unit for filtering special points in the current body surface point cloud, projecting the current body surface point cloud onto the infrared image based on a preset coordinate conversion relationship, and determining the target area according to the infrared image;

[0042] A point cloud registration unit for extracting the body surface point cloud in the target area for point cloud registration, calculating the difference between the current body surface point cloud and the reference body surface point cloud, and obtaining the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud;

[0043] An image rendering unit for rendering based on the current body surface point cloud to obtain a target image.

[0044] A two-modal based contour scanning imaging device includes: a memory and a processor;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the programs, each step of the foregoing two-modal based contour scanning imaging method.

[0047] A storage medium stores a computer program, and when the computer program is executed by a processor, it implements each step of the foregoing two-modal based contour scanning imaging method.

[0048] As can be seen from the above technical solution, for the two-modal contour scanning imaging method provided by the present invention, first, a two-dimensional image and an infrared image of the human body to be measured are acquired. Then, phase unwrapping is performed on the two-dimensional image; special points in the current body surface point cloud are filtered, and based on a preset coordinate conversion relationship, the current body surface point cloud is projected onto the infrared image; points in the body surface point cloud within the target area are extracted for point cloud registration, and the difference between the current body surface point cloud and the reference body surface point cloud is calculated to obtain the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud; rendering is performed based on the current body surface point cloud to obtain the target image. Based on ordinary three-dimensional imaging, the present invention adds an infrared image, increasing the data dimension from three-dimensional to four-dimensional, which can weaken the registration sliding effect and improve the speed and accuracy of registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0050] Figure 1 It is a flowchart of a two-modal contour scanning imaging method provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic structural diagram of an image acquisition system provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of an imaging effect provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic structural diagram of a two-modal contour scanning imaging device provided by an embodiment of the present invention;

[0054] Figure 5 It is a hardware structure block diagram of a two-modal contour scanning imaging device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0056] First, in combination with Figure 1An introduction is given to a two-modal based contour scanning imaging method provided in this embodiment, as Figure 1 shown. The method includes:

[0057] Step S01, obtaining a two-dimensional image and an infrared image of the human body to be measured.

[0058] Specifically, receive the two-dimensional image and the infrared image of the human body to be measured collected by the image acquisition system. As shown in Figure 2 the figure, the two-dimensional pattern projector projects sinusoidal stripes at a specific frequency onto the human body to be measured. Then, the contour scanning imaging system collects the two-dimensional image of the human body to be measured, and the infrared imaging system collects the infrared image of the human body to be measured under the infrared light source. The infrared imaging system can be an infrared camera. In addition, in order for the infrared imaging system to better collect the infrared image of the human body to be measured, a polarizer can be set in front of the infrared imaging system and the infrared light source to prevent the influence of reflection on the infrared imaging effect. A band-pass filter corresponding to the wavelength can also be added in front of the lens of the infrared imaging system to protect the infrared imaging system from the influence of external ambient light.

[0059] Step S02, performing phase unwrapping on the two-dimensional image.

[0060] Specifically, the GPU can be used to process the two-dimensional image and the infrared image. Copy the received two-dimensional image and infrared image into the GPU. The GPU performs phase unwrapping on the two-dimensional image to obtain the body surface point cloud. Among them, the process of phase unwrapping is as follows:

[0061] Using the fringe projection technology, project the sinusoidal stripes onto the object to be measured through the projection device. Using the phase shift method, project multiple phase shift patterns to mark the unique position. The light intensity formula of the sinusoidal stripes is as follows:

[0062]

[0063] where a(x,y) is the ambient light intensity, that is, the reflected light of the object to be measured, b(x,y) is the modulated light intensity, φ(x,y) is the initial phase information of the wavefront of the object to be measured, is the displacement amount, and (x,y) are the coordinates of the pixel points in the fringe pattern.

[0064] The two-dimensional image can be phase-unwrapped by the four-step phase shift method and the multi-frequency heterodyne method. The specific process is as follows:

[0065] Adopt the four-step phase shift method to obtain the phases to be modulated of the two-dimensional image. The phases to be modulated are as follows:

[0066] I1 = a + bcosφ;

[0067]

[0068] Based on the phase to be modulated, the principal value of the phase is obtained by solving through the arctangent function:

[0069]

[0070] Calculate by the multi-frequency heterodyne method to obtain a unique principal value within [0, 2π), calculate the phase difference of the sine gratings with different frequencies, convert the high-frequency phase into a low-frequency phase, so that the phase difference signal covers the entire field of view, and then obtain the absolute phase distribution of the two-dimensional image according to the phase difference.

[0071] Step S03, filter the special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on the pre-set coordinate transformation relationship, and determine the target area according to the infrared image.

[0072] Specifically, in order to improve the accuracy and speed of subsequent registration and the rendering efficiency during rendering, the special points that are meaningless in the body surface point cloud can be filtered by a geometry shader. Then, the remaining points in the current body surface point cloud are projected onto the infrared image based on the pre-set coordinate transformation relationship, and the target area of the body surface point cloud is determined according to the target area segmented from the infrared image. Among them, the coordinate transformation relationship is the coordinate transformation relationship between cameras determined by pre-calibrating the cameras using a checkerboard. For example: taking the blood vessel area on the infrared image as the target area, then segment the blood vessel area on the infrared image, project the body surface point cloud onto the infrared image, and determine the point cloud corresponding to the blood vessel area in the current body surface point cloud.

[0073] Step S04, extract the body surface point cloud within the target area for point cloud registration.

[0074] Specifically, after determining the target area in the current body surface point cloud, extract the body surface point cloud within the target area for point cloud registration. The process of point cloud registration can be performed on the GPU based on the ICP algorithm. For example: perform point cloud registration according to the point cloud of the blood vessel area of the patient to be measured. Project the body surface point cloud solved from the two-dimensional image to the area after blood vessel segmentation of the infrared image to obtain the real-time body surface target point cloud to be registered. During the positioning process, the benchmark of the body surface point cloud will be determined, and in the subsequent process, the real-time body surface point cloud will be registered with the benchmark body surface point cloud to calculate the rotation and translation relationship between the current body surface point cloud and the benchmark body surface point cloud. According to the rotation and translation relationship, the movement situation of the patient can be known.

[0075] Step S05, render based on the current body surface point cloud.

[0076] Specifically, use the multi-channel rendering + surface splash + point sprite rendering technology to directly render the current body surface point cloud to obtain the target image. Perform multi-channel rendering on the current body surface point cloud through three rendering channels. Figure 3It is the effect display of the rendered target image. The entire multi-channel rendering process adopts surface splatting. Multi-channel rendering is to render an object multiple times, and the results of each rendering process will be accumulated into the final rendering result. It is the overall architecture of the rendering method. Point sprite rendering is a method where a vertex is treated as a sprite. The advantage is that a single vertex can map a texture. What originally required four vertices to form a rectangle can now be completed with some vertices, reducing calculations and optimizing the rendering speed.

[0077] Surface splatting is a point rendering and texture filtering technique for rendering opaque and transparent surfaces directly from non-connected point clouds. Based on the screen area representation of the elliptical weighted average (EWA) filter, texture resampling is extended to irregularly spaced point samples. Surface splatting rendering assigns a Gaussian filtering kernel with a radius symmetry to each footprint, and reconstructs a continuous surface through the weighted average of the footprint data. Surface splatting rendering provides high-quality anisotropic texture filtering, hidden surface removal, edge anti-aliasing, and transparency with order independence. It belongs to a specific optimization method for rendering, but has a low running efficiency when rendering highly complex models.

[0078] Point sprite rendering technology is used to describe the process of a large number of particles moving on the screen. Graphics are composed of vertices, and texture mapping needs to be performed using vertices. A vertex is composed of a rectangle of 4 points. Point sprites can use only one vertex to draw a two-dimensional texture image to any position on the screen, saving the calculation of 3 points. Point sprite rendering technology reduces calculations and improves rendering efficiency, which is an optimization of the rendering speed.

[0079] The two-modal contour scanning imaging method provided in this embodiment first obtains a two-dimensional image and an infrared image of the human body to be measured. Then, phase unwrapping is performed on the two-dimensional image; special points in the body surface point cloud are filtered, and based on a pre-set coordinate transformation relationship, the body surface point cloud is projected onto the infrared image; points in the body surface point cloud within the target area are extracted for point cloud registration, and the difference between the current body surface point cloud and the reference body surface point cloud is calculated to obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud; rendering is performed based on the current body surface point cloud to obtain the target image. On the basis of ordinary three-dimensional imaging, this embodiment adds an infrared image dimension, increasing the data dimension from three-dimensional to four-dimensional, which can weaken the registration sliding effect and further improve the speed and accuracy of point cloud registration.

[0080] To make the point cloud registration result more accurate, after obtaining the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud by extracting the body surface point cloud within the target area for point cloud registration in step S04 of this embodiment, the following steps can also be executed:

[0081] Register the current body surface point cloud with the reference body surface point cloud to obtain a single-modal registration result. Perform weighted averaging on the point cloud registration result of the target region and the single-modal registration result to obtain the target registration result.

[0082] Specifically, use the current body surface point cloud and the reference body surface point cloud for single-modal point cloud registration, calculate the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud to obtain a single-modal registration result. Perform weighted averaging on the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud obtained based on the registration of the target region point cloud and the single-modal registration result to obtain the target registration result. The target registration result is used to represent the displacement of the patient.

[0083] In this embodiment, the single-modal registration result and the multi-modal registration result are combined, so that the obtained target registration result can more accurately represent the displacement of the patient during the treatment process.

[0084] Next, this embodiment introduces the process of step S03: filtering special points in the current body surface point cloud, projecting the current body surface point cloud onto the infrared image based on a pre-set coordinate transformation relationship, and determining the target region according to the infrared image:

[0085] First, in order to improve the accuracy and speed of subsequent registration and the rendering efficiency during rendering, special points in the current body surface point cloud can be filtered through a geometry shader, and the geometry shader filters points with coordinate values of (0, 0, 0). The process of filtering special points in the current body surface point cloud is as follows:

[0086] a. Obtain the video memory address of the vertex array saved on the GPU for rendering.

[0087] b. Assign values to the array in 1 in the function of phase unwrapping and reconstructing the point cloud.

[0088] c. Pass the vertex array in step a into the rendering pipeline and discard rasterization. Pass the incoming vertex coordinates to the geometry shader in the vertex shader of the pipeline. In the geometry shader, make a judgment. If the vertex coordinate value is (0, 0, 0, 1) (homogeneous coordinates in the shader), then discard the vertex; otherwise, emit the vertex.

[0089] d. Enable transform feedback before drawing the point cloud. After drawing, obtain the number of points in the point cloud, denoted as count. Since the vertices emitted by the geometry shader are tightly arranged in video memory, only the first size * 3 * count data in this video memory needs to be used for processing in the subsequent registration and rendering stages.

[0090] Then, based on the pre-set coordinate transformation relationship, project the body surface point cloud onto the infrared image. The process of the point cloud coordinate system describing the spatial positions of the camera and the object to be measured is as follows:

[0091] a. According to the transformation relationship between the point cloud coordinate system and the infrared camera coordinate system, convert the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system to obtain the infrared camera coordinates of each point.

[0092] Specifically, complete the transformation from the point cloud coordinate system to the infrared camera coordinate system through rigid body transformation. Through translation and rotation, only the spatial position and orientation of the figure are changed, and the shape of the object remains unchanged. Convert the coordinates of each point in the body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system through the formula as follows:

[0093] [x c y c z c T =R[x w y w z w T +t,

[0094] where, P w =[x w y w z w T , x w , y w , z w are respectively the three coordinate components of point P in the point cloud coordinate system w, x c , y c , z c are the three coordinate components of point P in the infrared camera coordinate system c, R and t are two transformation matrices; the origin of the infrared camera coordinate system is the optical center of the lens, the x and y axes are respectively parallel to both sides of the imaging plane, the z axis is the optical axis of the lens, the z axis is perpendicular to the imaging plane, the size of R is 3×3, and the size of t is 3×1.

[0095] b. According to the transformation relationship between the infrared camera coordinate system and the infrared image coordinate system, convert the infrared camera coordinates of each point from the infrared camera coordinate system to the infrared image coordinate system to obtain the infrared image coordinates of each point.

[0096] ​​​Specifically, the infrared image coordinate system is a planar coordinate system, representing the position of pixels in physical units, with the unit of mm. Among them, the origin of the infrared image coordinate system is the intersection point of the camera optical axis and the imaging plane, generally the midpoint of the imaging plane, and the x and y axes are parallel to both sides of the imaging plane respectively. The 3D-to-2D conversion is completed according to the perspective projection relationship. According to the principle of similar triangles, the infrared camera coordinates of each point are converted from the point cloud coordinate system to the infrared camera coordinate system through the following formula:

[0097]

[0098] Finally, the process of determining the target area based on the infrared image is as follows:

[0099] a. Perform Mask extraction on the infrared image. Invert the grayscale image of the obtained infrared image, converting white pixels to black and black pixels to white, and the same applies to pixels in other color value segments. Select a rough target area on the infrared image, distinguish the foreground and background according to the threshold segmentation method, create a mask, perform a bitwise AND operation on the mask and the original infrared image, retain the pixels in the filtered area, and set the pixels in the remaining areas to 0.

[0100] b. Extract the target area. Extract the target area for registration. For example, when the target area is a blood vessel, the blood vessel features can be extracted through a morphological algorithm. Use a line segment with a length of 10 and an orientation of i×15°, (0≤i<12) as the structural element b to cover blood vessels in all directions. Perform a morphological top-hat transformation on the image f through the following formula to extract the image details. The formula is as follows:

[0101]

[0102] Among them, h is the image after the top-hat transformation, represents the opening operation, represents the erosion operation, represents the dilation operation.

[0103] c. Image enhancement. Use the Contrast Limited Adaptive Histogram Equalization (CLAHE) method to enhance the image contrast. Divide the image into non-overlapping sub-blocks of equal size. Calculate the histogram of the sub-blocks, perform contrast limitation according to the clipping threshold clipLimit, evenly distribute the pixel distribution beyond the threshold to the probability density distribution, perform histogram equalization operation, and use bilinear interpolation to reduce the block effect after segmentation.

[0104] d. Image denoising. Use Gaussian blur to smooth the noise and reduce the influence of noise. Among them, the Gaussian kernel is: x, y are the pixel coordinates, and σ is the standard deviation.

[0105] Sharpen the image using the second-order Laplacian differential operator:

[0106] Since the image gray value is an integer between 0 and 255 and the x value is discrete, the formula needs to be extended from one-dimensional space to two-dimensional space. The process of deriving the formula from one-dimensional space to two-dimensional space is as follows:

[0107] The second derivative of a discontinuous function is:

[0108] f‘’(x) = f’(x + 1) - f’(x) = f(x + 1) + f(x - 1) - 2f(x).

[0109]

[0110] The Laplacian operator in two-dimensional space represents the function gain after a small change at a point in 4 degrees of freedom, where the directions are (1, 0), (-1, 0), (0, 1), (0, -1), and there are 8 degrees of freedom including the diagonal. The change method of this calculation process is expressed by the Laplacian convolution kernel, and the convolution kernel used is:

[0111] Perform Fourier transform on the infrared image: Among them, F(u, v) represents the frequency-domain image of the infrared image, f(x, y) represents the time-domain image of the infrared image, the width of the image is M, and the height is N.

[0112] Move the zero-frequency component to the center of the frequency-domain image and perform Gaussian low-pass filtering on the frequency-domain image of the infrared image: Among them, D0 is the cut-off frequency, which can be set to 2. The cut-off frequency can determine the frequency band width of the filter, that is, the smoothness. D(u, v) is the distance of the image point (u, v) from the frequency center, (u, v) is the coordinate of each point on the frequency-domain image, and the value ranges are [0, M - 1] and [0, N - 1] respectively. M and N are the widths of the image columns and rows respectively.

[0113] Perform inverse zero translation and two-dimensional discrete Fourier inverse transform on the filtered image to restore the image:

[0114]

[0115] e. Threshold segmentation, use a binary threshold to segment the restored image and convert the segmented image into a binary image. Among them, the white area represents the target area, and the rest of the area is set to black. If the target area is blood vessels, then the blood vessel area is the white area.

[0116] f. Denoise the target area image. According to the area {S i} of each connected area of the binary image, set the threshold Ts , the target area is screened to remove isolated noise points. If the target area is a blood vessel, the blood vessel is screened.

[0117] g. Determination of the point cloud of the target area. If the pixel value of the infrared image projected by a certain point in the body surface point cloud is 1, then this point belongs to the point cloud of the target area. After the results of all points are calculated, the point cloud of the target area can be extracted and the ICP algorithm is used for registration subsequently.

[0118] In this embodiment, the three-dimensional information is projected onto the plane of the IR image (infrared image), the perspective multi-point algorithm is used to combine the three-dimensional and two-dimensional images, the 3D / IR matrix is calculated, and the information of the three-dimensional body surface point cloud and the two-dimensional infrared image is associated.

[0119] Furthermore, this embodiment introduces the process of point cloud registration for the body surface point cloud in the target area in step S04. The ICP algorithm is used to register the point cloud on the GPU. The iterative closest point ICP algorithm mainly includes four stages:

[0120] (1) Sampling the point cloud data of the original target area. Using the sampling loss method based on the normal distribution of the points on the object surface, so that there are similar numbers of points in each normal direction, and the fine features of the point cloud of the target area are retained as much as possible.

[0121] (2) Determining the initial corresponding point sets P and Q. Using the point-to-plane algorithm, calculating the distance from the point to the tangent plane, and determining the reference point set P = {p1, p2,..., p n} and the data point set Q = {q1, q2,..., q n}.

[0122] (3) Removing the incorrect corresponding point set. Based on the constraint method of rigid motion consistency, using the principle that the corresponding points in the overlapping area of the measured object remain unchanged under rigid motion, and using the adjacent relationship between the corresponding point pairs as the consistency criterion, finding the corresponding nearest point in P for each point in Q to form a matching point pair (p, q). If ||p - q|| > ∈σ, then remove the point pair (p, q), where σ is the average value of the distances between all corresponding point pairs and ∈ is a given threshold.

[0123] (4) Solving for the coordinate transformation. Taking the sum of the Euclidean distances of all matching point pairs as the objective function to be solved. Using the singular value decomposition method (SVD) to find R and t to minimize the objective function. By constructing a least squares problem, the sum of the squares of the errors of the objective function reaches a minimum. The objective function is as follows:

[0124]

[0125] Finding the corresponding rotation matrix R and translation vector t, where, ω iRepresents the weight of each point, d represents the dimensions of x and y, and for the body surface point cloud, d = 3.

[0126] First, on the basis of fixing R, solve for the translation vector t. At this time, the objective function is:

[0127]

[0128] Take the partial derivative of t with respect to the current objective function to obtain: Among them, Is the centroid of the weighted average, Can be denoted as:

[0129] Secondly, after obtaining t, solve for R to get:

[0130] Derive the formula: Because, Is a scalar. According to the scalar property a = a T , it can be obtained that: Substitute to solve for R:

[0131] According to the properties of the matrix trace, it can be known that:

[0132] Define the covariance matrix S = XWY T , perform SVD decomposition on S to get: S = U∑V T , substitute to get: tr(WY T RX) = tr(∑V T RU). According to the properties of the orthogonal matrix and SVD decomposition, solve and calculate R = VU T .

[0133] Set the threshold of the iteration error to e. If E(R, t) > e, then according to R and t, convert to a new Q', continue to find the corresponding point pairs, and iterate like this until E(R, t) is less than the set threshold e.

[0134] Furthermore, this embodiment introduces the process of step S05, which is to render based on the current body surface point cloud. Use the multi-channel rendering + surface splash + point sprite rendering technology to directly render the current body surface point cloud to obtain the target image.

[0135] 1. The process of multi-channel rendering of the current body surface point cloud through three channels is as follows:

[0136] The first rendering channel: Enable depth testing, disable color writing, enable depth writing, disable blending, disable point sprites, adjust the depth of each fragment in the particle according to the particle radius, render the current body surface point cloud, and output a depth texture map. In theory, each pixel on the screen has to go through the calculation of the shader, and if there are objects with front and back occlusion, overlapping, blending, etc., a pixel may go through the calculation of multiple fragment shaders;

[0137] The second rendering channel: Disable depth testing, enable color writing, disable depth writing, enable blending, set the blending formula, use the depth values of each fragment in the first rendering channel to adjust the color value of each fragment, render the depth texture map, and output a color texture map;

[0138] The third rendering channel: Disable blending, draw a full-screen quadrilateral in the normalized device coordinate system, and paste the color texture map of the second rendering channel.

[0139] 2. The process of rendering through the surface splash technique:

[0140] Represent the object composed of points as a set of unconnected points {P k} that are irregularly distributed in three-dimensional space, which is related to the radially symmetric basis function r k and the coefficients representing the red, green, and blue color components respectively, and use w k as a generalized representation.

[0141] Select any point Q on the surface to establish a functional expression between Q and its surrounding extremely small range {Pk}. Consider the point Q and the points P k in the adjacent extremely small region as the corresponding coordinates u and u k in a two-dimensional plane, and define the function of this two-dimensional plane as the weighted sum of each point:

[0142] f c (u) = Σ k∈N w k r k (u - u k ),

[0143] Give the mapping relationship from the function plane to the screen x = m(u): R 2 → R 2 , which requires the following three steps:

[0144] (1) Transform f c (u) into a screen area to generate a continuous area signal:

[0145] g c (x) = (f c ⊙ m -1 )(x) = fc (m -1 (x))

[0146] Among them, ⊙ represents the connection of functions.

[0147] (2) Using the initial filter h for the band-limited screen area signal, the continuous function is calculated as:

[0148]

[0149] Among them, represents the convolution operation.

[0150] (3) Sampling the result of the continuous function, multiplying the continuous result by the pulse sequence j(x) to obtain the discrete result: g(x) = g' c (x)j(x).

[0151] (4) Reversely expanding the above relationship can inversely deduce the detailed expression:

[0152]

[0153] Among them, ρ k (x) is the resampling kernel. Independently transform and filter each basis function r k to construct the resampling kernel ρ k , and the process of accumulating these kernels in the screen area is surface splashing.

[0154] In addition, ρ k (x) can be further simplified by replacing m(u) with its local radiation estimate at point u k :

[0155]

[0156] Among them, x k = m(u k ), and the Jacobian matrix is further calculated as:

[0157]

[0158] Among them, represents the transformed basis function. Although the texture function is defined on an irregular grid, the resampling kernel ρ k (x) of the screen area can still be expressed in the form of the convolution of the transformed basis function r' k and the low-pass filter kernel h.

[0159] 3. Rendering process of point sprite technology:

[0160] There is a built-in read-only variable gl_PointCoord in the fragment shader, which interpolates texture coordinates at vertices.

[0161] There is a built-in read-only variable gl_PointSize in the vertex shader, which is used to control the final raster size of the point, and it is a pixel value. The point size can be determined according to the point distance, and the point distance transformation formula is:

[0162]

[0163] Among them, d is the distance from the point to the observation point, and a, b, c are the parameters of the quadratic equation. They can be stored as unified values or set as constants in the vertex shader. a controls the constant part of the final value, b controls the linear change of the final value with distance, and c controls the quadratic relationship of the final value with distance transformation.

[0164] Next, the two-modal based contour scanning imaging device provided by the embodiments of the present invention will be described. The two-modal based contour scanning imaging device described below can be correspondingly referred to the two-modal based contour scanning imaging method described above.

[0165] First, in combination with Figure 4 , the two-modal based contour scanning imaging device will be introduced. As Figure 4 shown, the two-modal based contour scanning imaging device may include:

[0166] An image acquisition unit 100, configured to acquire a two-dimensional image and an infrared image of a human body to be measured;

[0167] A point cloud acquisition unit 200, configured to perform phase unwrapping on the two-dimensional image to obtain a current body surface point cloud;

[0168] A point cloud processing unit 300, configured to filter special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on a preset coordinate conversion relationship, and determine a target area according to the infrared image;

[0169] A point cloud registration unit 400, configured to extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud;

[0170] An image rendering unit 500, configured to perform rendering based on the current body surface point cloud to obtain a target image.

[0171] The two-modal based contour scanning imaging device provided by the embodiments of the present invention can be applied to a two-modal based contour scanning imaging device. Figure 5 Shows the hardware structure block diagram of the two-modal based contour scanning imaging device. Refer to Figure 5, the hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0172] In the embodiments of the present invention, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete mutual communication through the communication bus 4;

[0173] The processor 1 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0174] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0175] Among them, the memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the foregoing two-modal contour scanning imaging scheme.

[0176] The embodiments of the present invention also provide a storage medium, which can store a program suitable for execution by a processor, and the program is used to implement each processing flow in the foregoing two-modal contour scanning imaging scheme.

[0177] Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The various embodiments in this specification are described in a progressive manner, and the key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0178] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A two-modal based contour scanning imaging method, characterized in that The method includes: Obtaining a two-dimensional image and an infrared image of the human body to be measured; Performing phase unwrapping on the two-dimensional image to obtain the current body surface point cloud; Filtering special points in the current body surface point cloud, and based on a preset coordinate transformation relationship, projecting the current body surface point cloud onto the infrared image to determine the target area according to the infrared image; Extracting the body surface point cloud within the target area for point cloud registration, calculating the difference between the current body surface point cloud and the reference body surface point cloud, and obtaining the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud; Rendering based on the current body surface point cloud to obtain the target image; Among them, the process of extracting the body surface point cloud within the target area for point cloud registration and calculating the difference between the current body surface point cloud and the reference body surface point cloud includes: Projecting the current body surface point cloud onto the target area to obtain the real-time body surface target point cloud; Registering the real-time body surface target point cloud with a pre-determined reference body surface point cloud to obtain the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud.

2. The two-modal contour scanning imaging method according to claim 1, characterized in that The process of rendering based on the current body surface point cloud to obtain the target image includes: Using multi-channel rendering + surface splashing + point sprite rendering technology to render the current body surface point cloud.

3. The two-modal contour scanning imaging method according to claim 2, wherein The process of using multi-channel rendering to render the current body surface point cloud includes: Performing multi-channel rendering on the current body surface point cloud through three rendering channels; Among them, the processes of rendering by the three rendering channels are as follows: The first rendering channel: enabling depth testing, disabling color writing, enabling depth writing, disabling blending, disabling point sprites, adjusting the depth of each fragment in the particle according to the particle radius, rendering the current body surface point cloud, and outputting a depth texture map; The second rendering channel: disabling depth testing, enabling point sprites, enabling color writing, disabling depth writing, enabling blending, setting the blending formula, using the depth values of each fragment in the first rendering channel to adjust the color values of each fragment, rendering the depth texture map, and outputting a color texture map; The third rendering channel: disabling blending, drawing a full-screen quadrilateral in the normalized device coordinate system, and pasting the color texture map of the second rendering channel.

4. The two-modal contour scanning imaging method according to claim 1, wherein Performing phase unwrapping on the two-dimensional image by the four-step phase-shifting method and the multi-frequency heterodyne method, the process includes: Obtaining the to-be-modulated phases of the two-dimensional image by the four-step phase-shifting method; Based on the to-be-modulated phases, solving through the arctangent function to obtain the principal value of the phase; Performing calculations by the multi-frequency heterodyne method to obtain a unique principal value within [0, 2π), calculating the phase difference of sine gratings with different frequencies, converting the high-frequency phase into a low-frequency phase, enabling the phase difference signal to cover the entire field of view, and then obtaining the absolute phase distribution of the two-dimensional image according to the phase difference.

5. The two-modal contour scanning imaging method according to claim 1, wherein The process of projecting the current body surface point cloud onto the infrared image based on a preset coordinate transformation relationship includes: According to the conversion relationship between the point cloud coordinate system and the infrared camera coordinate system, converting the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system to obtain the infrared camera coordinates of each point, where the origin of the infrared camera coordinate system is the optical center of the lens, the x and y axes are respectively parallel to both sides of the imaging plane, and the z axis is the optical axis of the lens; According to the conversion relationship between the infrared camera coordinate system and the infrared image coordinate system, the infrared camera coordinates of each point are converted from the infrared camera coordinate system to the infrared image coordinate system to obtain the infrared image coordinates of each point. Among them, the origin of the infrared image coordinate system is the midpoint of the imaging plane, and the x and y axes are parallel to both sides of the imaging plane respectively.

6. The two-modal based contour scanning imaging method according to claim 5, wherein The process of converting the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system includes: Converting the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system through a formula. The formula is as follows: [x c y c z c T = R[x w y w z w T + t,​​ where x w , y w , z w are the three coordinate components of point P in the point cloud coordinate system w, and x c , y c , z c are the three coordinate components of point P in the infrared camera coordinate system c, and R and t are two transformation matrices; The process of converting the infrared camera coordinates of each point from the infrared camera coordinate system to the infrared image coordinate system includes: Based on the principle of similar triangles, converting the infrared camera coordinates of each point from the point cloud coordinate system to the infrared camera coordinate system through a formula. The formula is as follows:

7. The two-modal based contour scanning imaging method according to any one of claims 1-6, characterized in that After extracting the body surface point cloud in the target area for point cloud registration and calculating the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud, it further includes: Registering the current body surface point cloud with the reference body surface point cloud to obtain a single-modal registration result; Performing weighted averaging on the target area point cloud registration result and the single-modal registration result to obtain a target registration result.

8. A two-modal contour scanning imaging device, characterized in that, It further includes: An image acquisition unit for acquiring a two-dimensional image and an infrared image of the human body to be measured; A point cloud acquisition unit for performing phase unwrapping on the two-dimensional image to obtain the current body surface point cloud; A point cloud processing unit for filtering special points in the current body surface point cloud, projecting the current body surface point cloud onto the infrared image based on a pre-set coordinate conversion relationship, and determining the target area according to the infrared image; A point cloud registration unit for extracting the body surface point cloud in the target area for point cloud registration, calculating the difference between the current body surface point cloud and the reference body surface point cloud, and obtaining the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud; An image rendering unit for rendering based on the current body surface point cloud to obtain a target image; Among them, the process in which the point cloud registration unit extracts the body surface point cloud in the target area for point cloud registration and calculates the difference between the current body surface point cloud and the reference body surface point cloud includes: Projecting the current body surface point cloud onto the target area to obtain a real-time body surface target point cloud; Registering the real-time body surface target point cloud with the pre-determined reference body surface point cloud to obtain the rotation and translation relationships between the current body surface point cloud and the reference body surface point cloud.

9. A two-modal based contour scanning imaging device, characterized in that It includes: A memory and a processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the two-modal based contour scanning imaging method according to any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the two-modal based contour scanning imaging method according to any one of claims 1-7.

Images