Contour scanning imaging method and device based on two modes, equipment and storage medium

Abstract

The invention discloses a contour scanning imaging method, device and equipment based on two modalities and a storage medium, and the method comprises the steps: firstly, obtaining a two-dimensional image and an infrared image of a to-be-detected human body, and then carrying out the phase unwrapping of the two-dimensional image; then special points in the current body surface point cloud are filtered, and the current body surface point cloud is projected to an infrared image based on a coordinate conversion relation; points in the current body surface point cloud in the target area are extracted for point cloud registration, the difference between the current body surface point cloud and the reference body surface point cloud is calculated, and the rotation and translation relation between the current body surface point cloud and the reference body surface point cloud is obtained; and finally, rendering based on the current body surface point cloud to obtain a target image. On the basis of common three-dimensional imaging, the infrared image is added, the data dimension is increased from three dimensions to four dimensions, the registration sliding effect can be weakened, and the registration speed and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scanning technology, and more specifically, to a two-modality-based contour scanning imaging method, device, equipment and storage medium. Background Art

[0002] Body surface three-dimensional information refers to the three-dimensional shape, size and position information of the human body or object surface obtained through three-dimensional scanning technology. The acquisition of body surface three-dimensional information mainly relies on three-dimensional scanning technology, which can quickly and accurately obtain three-dimensional data of the surface of an object through non-contact measurement. Three-dimensional scanning technology is widely used in the fields of medicine, industrial design, forensic medicine, film and television special effects, etc. In the medical field, it can be used in the surface guidance system in radiotherapy to collect the patient's body surface three-dimensional information before and during treatment, so as to accurately position the patient before and during radiotherapy, thereby achieving precise radiotherapy of the affected part.

[0003] At present, in order to obtain three-dimensional information of the human body surface, the scanning imaging system projects a pattern on the human body surface, where the projected pattern includes but is not limited to one-dimensional line structured light and two-dimensional patterns. During the three-dimensional body surface scanning process, the camera captures the pattern projected on the human body surface. If the surface is flat, the pattern captured by the camera is similar to the original pattern projected by the projector; if the surface is uneven, the pattern captured by the camera will be distorted. At this time, it is necessary to calculate the depth information of the target point based on the spatial position of the camera and the projector.

[0004] The core of the 3D scanning system is to compare the human body surface with the reference surface to detect spatial positioning deviations. In the field of computer vision, the iterative closest point (ICP) algorithm is a common point cloud registration method. Among them, the ICP algorithm continuously iterates to reduce the distance between two point clouds. However, when the ICP algorithm registers objects with translational or rotational symmetry, it will cause a registration sliding effect. For example, areas such as the chest and flat abdomen of the human body are more likely to appear, resulting in low speed and accuracy of point cloud registration. Summary of the invention

[0005] In view of this, the present invention provides a contour scanning imaging method, device, equipment and storage medium based on two modes, which are used to solve the problem that the existing three-dimensional scanning imaging method uses the ICP algorithm to align all points in two human point clouds, which is prone to a registration sliding effect, resulting in low speed and accuracy of human point cloud registration.

[0006] To achieve the above objectives, the proposed solution is as follows:

[0007] A contour scanning imaging method based on two modes, the method comprising:

[0008] Acquire a two-dimensional image and an infrared image of the human body to be tested;

[0009] Perform phase unwrapping on the two-dimensional image to obtain the current body surface point cloud;

[0010] Filter the special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on a preset coordinate transformation relationship, and determine the target area according to the infrared image;

[0011] Extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud;

[0012] Rendering is performed based on the current body surface point cloud to obtain the target image.

[0013] Preferably, the process of rendering based on the current body surface point cloud to obtain the target image includes:

[0014] Use multi-channel rendering + surface splash + point sprite rendering technology to render the current surface point cloud.

[0015] Preferably, the process of rendering the current body surface point cloud using multi-channel rendering includes:

[0016] Perform multi-channel rendering of the current body surface point cloud through three rendering channels;

[0017] Among them, the rendering process of the three rendering channels is as follows:

[0018] First rendering pass: Enable depth test, disable color writing, enable depth writing, disable blending, disable point sprites, adjust the depth of each fragment in the particle according to the particle radius, render the current surface point cloud, and output the depth texture map;

[0019] Second rendering channel: turn off depth test, turn on point sprite, enable color writing, turn off depth writing, enable blending, set blending formula, use the depth value of each fragment in the first rendering channel to adjust the color value of each fragment, render the depth texture map, and output the color texture map;

[0020] Pass 3: Turn off blending, draw a full-screen quad in normalized device coordinates, and apply the color texture map from Pass 2.

[0021] Preferably, the two-dimensional image is phase-unwrapped by a four-step phase shift method and a multi-frequency heterodyne method, and the process includes:

[0022] A four-step phase shift method is used to obtain each phase to be modulated of a two-dimensional image;

[0023] Based on the phase to be modulated, the main phase value is obtained by solving the inverse tangent function;

[0024] Through the multi-frequency heterodyne method, a unique principal value is obtained between [0,2π), and the phase difference of sinusoidal gratings of different frequencies is calculated to convert the high-frequency phase into a low-frequency phase so that the phase difference signal covers the entire field of view. Then, the absolute phase distribution of the two-dimensional image is obtained based on the phase difference.

[0025] Preferably, based on a preset coordinate transformation relationship, the process of projecting the current body surface point cloud onto the infrared image includes:

[0026] According to the conversion relationship between the point cloud coordinate system and the infrared camera coordinate system, the coordinates of each point in the current body surface point cloud are converted from the point cloud coordinate system to the infrared camera coordinate system to obtain the infrared camera coordinates of each point, where the origin of the infrared camera coordinate system is the optical center of the lens, the x and y axes are parallel to the two sides of the imaging plane, and the z axis is the optical axis of the lens;

[0027] According to the conversion relationship between the infrared camera coordinate system and the infrared image coordinate system, the infrared camera coordinates of each point are converted from the infrared camera coordinate system to the infrared image coordinate system to obtain the infrared image coordinates of each point, where the origin of the infrared image coordinate system is the midpoint of the imaging plane, and the x and y axes are parallel to the two sides of the imaging plane, respectively.

[0028] Preferably, the process of converting the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system includes:

[0029] The coordinates of each point in the current body surface point cloud are converted from the point cloud coordinate system to the infrared camera coordinate system through the formula. The formula is as follows:

[0030] [x c y c z c ] T =R[x w y w z w ] T +t,

[0031] Among them, x w ,y w ,z w They are the three coordinate components of point P in the point cloud coordinate system w, x c ,y c ,z c are the three coordinate components of point P in the infrared camera coordinate system c, R and t are two transformation matrices;

[0032] The process of converting the infrared camera coordinates of each point from the infrared camera coordinate system to the infrared image coordinate system includes:

[0033] Based on the principle of similar triangles, the infrared camera coordinates of each point are converted from the point cloud coordinate system to the infrared camera coordinate system through the formula. The formula is as follows:

[0034]

[0035] Preferably, after extracting the body surface point cloud in the target area for point cloud registration and calculating the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud, the method further includes:

[0036] Registering the current body surface point cloud with the reference body surface point cloud to obtain a single-modality registration result;

[0037] The target area point cloud registration result and the single modality registration result are weighted averaged to obtain the target registration result.

[0038] A contour scanning imaging device based on two modes, further comprising:

[0039] An image acquisition unit, used to acquire a two-dimensional image and an infrared image of a human body to be tested;

[0040] A point cloud acquisition unit is used to perform phase unwrapping on the two-dimensional image to obtain a current body surface point cloud;

[0041] A point cloud processing unit is used to filter special points in the current body surface point cloud, and based on a preset coordinate conversion relationship, project the current body surface point cloud onto an infrared image, and determine a target area according to the infrared image;

[0042] A point cloud registration unit is used to extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud;

[0043] The image rendering unit is used to render based on the current body surface point cloud to obtain a target image.

[0044] A contour scanning imaging device based on two modes, comprising: a memory and a processor;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the program and the various steps of the aforementioned two-modality-based contour scanning imaging method.

[0047] A storage medium stores a computer program, which, when executed by a processor, implements each step of the aforementioned contour scanning imaging method based on two modalities.

[0048] It can be seen from the above technical solutions that the contour scanning imaging method based on two modes provided by the present invention first obtains a two-dimensional image and an infrared image of the human body to be measured, and then performs phase expansion on the two-dimensional image; filters the special points in the current body surface point cloud, and projects the current body surface point cloud to the infrared image based on a preset coordinate transformation relationship; extracts points in the body surface point cloud in the target area for point cloud registration, calculates the difference between the current body surface point cloud and the reference body surface point cloud, and obtains the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud; renders based on the current body surface point cloud to obtain the target image. Based on ordinary three-dimensional imaging, the present invention adds an infrared image, increases the data dimension from three dimensions to four dimensions, can reduce the registration sliding effect, and improve the speed and accuracy of registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A flow chart of a contour scanning imaging method based on two modes provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the structure of an image acquisition system provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of an imaging effect provided by an embodiment of the present invention;

[0053] Figure 4 A schematic structural diagram of a dual-modality contour scanning imaging device provided by an embodiment of the present invention;

[0054] Figure 5 A hardware structure block diagram of a dual-modality contour scanning imaging device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] First, combine Figure 1A contour scanning imaging method based on two modes provided in this embodiment is introduced. Figure 1 As shown, the method includes:

[0057] Step S01, obtaining a two-dimensional image and an infrared image of a human body to be tested.

[0058] Specifically, the two-dimensional image and infrared image of the human body to be tested collected by the image acquisition system are received. Figure 2 As shown, the two-dimensional pattern projector uses a specific frequency to project sinusoidal stripes onto the human body to be tested. Then, the contour scanning imaging system collects the two-dimensional image of the human body to be tested, and the infrared imaging system collects the infrared image of the human body to be tested under the infrared light source. The infrared imaging system can be an infrared camera. In addition, in order for the infrared imaging system to better collect the infrared image of the human body to be tested, a polarizer can be set in front of the infrared imaging system and the infrared light source to prevent the influence of reflection on the infrared imaging effect. A bandpass filter of the corresponding wavelength can also be added in front of the infrared imaging system lens to protect the infrared imaging system from the influence of external ambient light.

[0059] Step S02: performing phase unwrapping on the two-dimensional image.

[0060] Specifically, the two-dimensional image and the infrared image can be processed by the GPU, the received two-dimensional image and the infrared image can be copied to the GPU, and the GPU performs phase unwrapping on the two-dimensional image to obtain the body surface point cloud. The phase unwrapping process is as follows:

[0061] Using fringe projection technology, the sinusoidal fringe is projected onto the object to be measured through a projection device, and using the phase shift method, multiple phase shift groups are projected to mark the unique position. The light intensity formula of the sinusoidal fringe is as follows:

[0062]

[0063] Among them, a(x,y) is the ambient light intensity, that is, the reflected light of the object to be measured, b(x,y) is the modulated light intensity, φ(x,y) is the initial phase information of the wavefront of the object to be measured, is the displacement, and (x,y) is the coordinate of the pixel in the fringe pattern.

[0064] The phase unwrapping of the two-dimensional image can be performed by the four-step phase shift method and the multi-frequency heterodyne method. The specific process is as follows:

[0065] The four-step phase shift method is used to obtain the phases to be modulated of the two-dimensional image. The phases to be modulated are as follows:

[0066] I1=a+bcosφ;

[0067]

[0068] Based on the phase to be modulated, the main phase value is obtained by solving the inverse tangent function:

[0069]

[0070] Through the multi-frequency heterodyne method, a unique principal value is obtained between [0,2π), and the phase difference of sinusoidal gratings of different frequencies is calculated to convert the high-frequency phase into a low-frequency phase so that the phase difference signal covers the entire field of view. Then, the absolute phase distribution of the two-dimensional image is obtained based on the phase difference.

[0071] Step S03: filter the special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on a preset coordinate conversion relationship, and determine the target area according to the infrared image.

[0072] Specifically, in order to improve the accuracy and speed of subsequent registration and the rendering efficiency during rendering, the geometry shader can be used to filter out meaningless special points in the body surface point cloud. Then, the remaining points in the current body surface point cloud are projected onto the infrared image according to a preset coordinate transformation relationship, and the target area of ​​the body surface point cloud is determined according to the target area segmented from the infrared image. Among them, the coordinate transformation relationship is the coordinate transformation relationship between cameras determined by pre-calibrating the camera using a checkerboard. For example, if the vascular area on the infrared image is used as the target area, the vascular area on the infrared image is segmented, the body surface point cloud is projected onto the infrared image, and the point cloud corresponding to the vascular area in the current body surface point cloud is determined.

[0073] Step S04: extracting the body surface point cloud within the target area and performing point cloud registration.

[0074] Specifically, after determining the target area in the current body surface point cloud, the body surface point cloud in the target area is extracted for point cloud registration. The process of point cloud registration can be performed on the GPU based on the ICP algorithm. For example: point cloud registration is performed based on the point cloud of the human blood vessel area to be measured. The body surface point cloud calculated from the two-dimensional image is projected to the area after the blood vessel segmentation of the infrared image to obtain the real-time point cloud of the body surface target that needs to be registered. The body surface point cloud reference will be determined during the positioning process. In the subsequent process, the real-time body surface point cloud will be registered with the reference body surface point cloud, and the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud is calculated. The patient's movement can be known based on the rotation and translation relationship.

[0075] Step S05: Rendering based on the current body surface point cloud.

[0076] Specifically, the current body surface point cloud is directly rendered using multi-channel rendering + surface splash + point sprite rendering technology to obtain the target image. The current body surface point cloud is multi-channel rendered through three rendering channels. Figure 3The effect of the rendered target image is shown. The entire multi-pass rendering process uses surface splashing. Multi-pass rendering is to render an object multiple times, and the results of each rendering process will be accumulated to the final rendering result. It is the overall architecture of the rendering method. Point sprite rendering is a vertex that is treated as a sprite. The advantage is that a vertex can be pasted with a texture. The rectangle originally composed of four vertices can now be completed by a few vertices, reducing calculations and optimizing rendering speed.

[0077] Surface splash is a point rendering and texture filtering technique that renders opaque and transparent surfaces directly from non-connected point clouds. Texture resampling is extended to irregularly spaced point samples based on the screen area representation of the Elliptical Weighted Average (EWA) filter. Surface splash rendering assigns a radius-symmetric Gaussian filter kernel to each footprint and reconstructs the continuous surface by weighted averaging the data of each footprint. Surface splash rendering provides high-quality anisotropic texture filtering, hidden surface removal, edge anti-aliasing, and order-independent transparency. It is a specific optimization method for rendering, but it has low operating efficiency when drawing highly complex models.

[0078] Point sprite rendering technology is used to describe the process of a large number of particles moving on the screen. Graphics are composed of vertices, and vertices need to be used for texture mapping. Vertices are composed of rectangles of 4 points. Point sprites can use only one vertex to draw a 2D texture image to any position on the screen, saving the calculation of 3 points. Point sprite rendering technology reduces calculations, improves rendering efficiency, and optimizes rendering speed.

[0079] The contour scanning imaging method based on two modes provided in this embodiment first obtains a two-dimensional image and an infrared image of the human body to be measured, and then performs phase unfolding on the two-dimensional image; filters the special points in the body surface point cloud, and projects the body surface point cloud to the infrared image based on a preset coordinate transformation relationship; extracts points in the body surface point cloud in the target area for point cloud registration, calculates the difference between the current body surface point cloud and the reference body surface point cloud, and obtains the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud; renders based on the current body surface point cloud to obtain the target image. This embodiment adds an infrared image dimension on the basis of ordinary three-dimensional imaging, and increases the data dimension from three dimensions to four dimensions, which can reduce the registration sliding effect and further improve the speed and accuracy of point cloud registration.

[0080] In order to make the point cloud registration result more accurate, in step S04, this embodiment extracts the body surface point cloud in the target area for point cloud registration, and after obtaining the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud, the following steps may be performed:

[0081] The current body surface point cloud is registered with the reference body surface point cloud to obtain the single-modality registration result. The target area point cloud registration result and the single-modality registration result are weighted averaged to obtain the target registration result.

[0082] Specifically, the current body surface point cloud and the reference body surface point cloud are used for single-modality point cloud registration, and the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud is calculated to obtain the single-modality registration result. The rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud obtained based on the target area point cloud registration is weighted averaged with the single-modality registration result to obtain the target registration result. The target registration result is used to represent the patient's displacement.

[0083] This embodiment combines the single-modality registration result with the multi-modality registration result, so that the obtained target registration result can more accurately represent the displacement of the patient during the treatment process.

[0084] Next, this embodiment introduces the process of filtering the special points in the current body surface point cloud in step S03, projecting the current body surface point cloud onto the infrared image based on a preset coordinate transformation relationship, and determining the target area according to the infrared image:

[0085] First, in order to improve the accuracy and speed of subsequent registration and the rendering efficiency during rendering, the special points in the current body surface point cloud can be filtered through the geometry shader. The geometry shader filters the points with coordinate values ​​of (0,0,0). The process of filtering the special points in the current body surface point cloud is as follows:

[0086] a. Get the video memory address of the vertex array used for rendering stored on the GPU.

[0087] b. Assign the array in 1 to the function of solving the phase and reconstructing the point cloud.

[0088] c. Pass the vertex array in step a into the rendering pipeline and discard rasterization. In the vertex shader of the pipeline, pass the passed vertex coordinates to the geometry shader. Make a judgment in the geometry shader. If the vertex coordinate value is (0,0,0,1) (all are homogeneous coordinates in the shader), discard the vertex; otherwise, emit the vertex.

[0089] d. Enable transform feedback before drawing the point cloud. After drawing, get the number of points in the point cloud, recorded as count. Since the vertices emitted by the geometry shader are densely arranged in the video memory, the subsequent registration and rendering stages only need to use the first size*3*count data in this video memory for processing.

[0090] Then, based on the preset coordinate transformation relationship, the body surface point cloud is projected onto the infrared image. The point cloud coordinate system describes the spatial position of the camera and the object to be measured as follows:

[0091] a. According to the conversion relationship between the point cloud coordinate system and the infrared camera coordinate system, the coordinates of each point in the current body surface point cloud are converted from the point cloud coordinate system to the infrared camera coordinate system to obtain the infrared camera coordinates of each point.

[0092] Specifically, the transformation from the point cloud coordinate system to the infrared camera coordinate system is completed through rigid body transformation. Through translation and rotation, only the spatial position and orientation of the image are changed, and the shape of the object is not changed. The coordinates of each point in the surface point cloud are converted from the point cloud coordinate system to the infrared camera coordinate system through the formula, which is as follows:

[0093] [x c y c z c ] T =R[x w y w z w ] T +t,

[0094] Among them, P w =[x w y w z w ] T , x w ,y w ,z w They are the three coordinate components of point P in the point cloud coordinate system w, x c ,y c ,z c are the three coordinate components of point P in the infrared camera coordinate system c, R and t are two transformation matrices; the origin of the infrared camera coordinate system is the optical center of the lens, the x and y axes are parallel to the two sides of the imaging plane respectively, the z axis is the optical axis of the lens, and the z axis is perpendicular to the imaging plane. The size of R is 3×3, and the size of t is 3×1.

[0095] b. According to the conversion relationship between the infrared camera coordinate system and the infrared image coordinate system, the infrared camera coordinates of each point are converted from the infrared camera coordinate system to the infrared image coordinate system to obtain the infrared image coordinates of each point.

[0096] Specifically, the infrared image coordinate system is a plane coordinate system, and the position of pixels is expressed in physical units, in units of mm. Among them, the origin of the infrared image coordinate system is the intersection of the camera optical axis and the imaging plane, which is generally the midpoint of the imaging plane, and the x and y axes are parallel to the two sides of the imaging plane, respectively. The conversion from 3D to 2D is completed according to the perspective projection relationship. According to the principle of similar triangles, the infrared camera coordinates of each point are converted from the point cloud coordinate system to the infrared camera coordinate system through the formula, and the formula is as follows:

[0097]

[0098] Finally, the process of determining the target area based on the infrared image is as follows:

[0099] a. Perform mask extraction on the infrared image, invert the color of the obtained infrared image grayscale image, convert white pixels to black, black pixels to white, and the same applies to pixels in other color value segments. Select a rough target area on the infrared image, distinguish the foreground and background according to the threshold segmentation method, make a mask, perform a bitwise AND operation on the mask and the original infrared image, retain the pixels in the filtered area, and set the pixels in the remaining area to 0.

[0100] b. Extract the target area. Extract the target area for registration. For example, when the target area is a blood vessel, the blood vessel features can be extracted through a morphological algorithm. Use a line segment with a length of 10 and a direction of i×15° (0≤i<12) as the structural element b to cover blood vessels in all directions. Perform a morphological top-hat transformation on the image f to extract image details using the following formula:

[0101]

[0102] Among them, h is the image after top-hat transformation, represents the opening operation, represents the corrosion operation, Represents an expansion operation.

[0103] c. Image enhancement, using the contrast-limited adaptive histogram equalization (CLAHE) method to enhance the image contrast. Divide the image into non-overlapping sub-blocks of equal size. Calculate the histogram of the sub-block, limit the contrast according to the clipping threshold clipLimit, evenly distribute the pixels exceeding the threshold to the probability density distribution, perform an equalization operation on the histogram, and use bilinear interpolation to reduce the blocking effect after segmentation.

[0104] d. Image denoising: Use Gaussian blur to smooth out the noise and reduce the impact of noise. The Gaussian kernel is: x, y are the pixel coordinates, and σ is the standard deviation.

[0105] Sharpen the image using the Laplace second-order derivative operator:

[0106] Since the grayscale value of the image is an integer between 0 and 255, and the value of x is discrete, the formula needs to be generalized from one-dimensional space to two-dimensional space. The process of deriving the formula from one-dimensional space to two-dimensional space is as follows:

[0107] The second derivative of a discontinuous function is:

[0108] f''(x)=f'(x+1)-f'(x)=f(x+1)+f(x-1)-2f(x).

[0109]

[0110] The Laplace operator in two-dimensional space represents the function gain of a point after a small change in 4 degrees of freedom, where the directions are (1,0), (-1,0), (0,1), (0,-1), and 8 degrees of freedom in total including the diagonal. The Laplace convolution kernel is used to express the change in the calculation process. The convolution kernel used is:

[0111] Take a Fourier transform of the infrared image: Among them, F(u,v) represents the frequency domain image of the infrared image, f(x,y) represents the time domain image of the infrared image, the image width is M, and the height is N.

[0112] Move the zero-frequency component to the center of the frequency domain image and perform Gaussian low-pass filtering on the frequency domain image of the infrared image: Among them, D0 is the cutoff frequency, which can be set to 2. The cutoff frequency can determine the bandwidth of the filter, that is, the degree of smoothness. D(u,v) is the distance between the image point (u,v) and the frequency center. (u,v) is the coordinate of each point on the frequency domain image. The value range is [0,M-1], [0,N-1], respectively. M and N are the widths of the image columns and rows, respectively.

[0113] Perform inverse zero shift and two-dimensional discrete Fourier inverse transform on the filtered image to restore the image:

[0114]

[0115] e. Threshold segmentation: Use the binary threshold to segment the restored image and convert the segmented image into a binary image. The white area represents the target area, and the rest of the area is set to black. If the target area is a blood vessel, the blood vessel area is the white area.

[0116] f. Target area image denoising, based on the area of ​​each connected region of the binary image {S i}, set the threshold Ts , filter the target area and remove isolated noise points. If the target area is a blood vessel, filter the blood vessel.

[0117] g. The point cloud of the target area is determined. If the pixel value of the infrared image projected onto a point in the body surface point cloud is 1, then the point belongs to the point cloud of the target area. After the results of all points are calculated, the point cloud of the target area can be extracted and subsequently registered using the ICP algorithm.

[0118] This embodiment projects three-dimensional information onto the IR image (infrared image) plane, uses a perspective multi-point algorithm to combine three-dimensional and two-dimensional images, calculates a 3D / IR matrix, and associates the three-dimensional body surface point cloud with the two-dimensional infrared image information.

[0119] Furthermore, this embodiment introduces the process of performing point cloud registration on the body surface point cloud in the target area in step S04, and uses the ICP algorithm on the GPU to register the point cloud, wherein the iterative closest point ICP algorithm is mainly divided into four stages:

[0120] (1) The point cloud data of the original target area is sampled, and the sampling method is used based on the normal distribution of the points on the surface of the object, so that each normal direction has a similar number of points, and the subtle features of the point cloud of the target area are retained as much as possible.

[0121] (2) Determine the initial corresponding point set P and Q. Use the point-to-plane algorithm to calculate the distance from the point to the tangent plane and determine the reference point set P = {p1, p2, ..., p n} and data point set Q = {q1,q2,...,q n}.

[0122] (3) Remove the wrong corresponding point set. Based on the constraint method of rigid motion consistency, the principle of rigid motion keeping the corresponding points in the overlapping area of ​​the measured object unchanged is used. The adjacent relationship between corresponding point pairs is used as the consistency criterion to find the corresponding nearest point in P for each point in Q to form a matching point pair (p, q). If ||pq||>∈σ, then remove the point pair (p, q), where σ is the mean distance between all corresponding point pairs and ∈ is a given threshold.

[0123] (4) Solve the coordinate transformation and take the sum of the Euclidean distances of all matching point pairs as the objective function to be solved. Use the singular value decomposition (SVD) method to find R and t to minimize the objective function. By constructing the least squares problem, the sum of squares of the objective function error is minimized. The objective function is as follows:

[0124]

[0125] Find the corresponding rotation matrix R and translation vector t, where ω iRepresents the weight of each point, d represents the dimensions of x and y, and the surface point cloud d=3.

[0126] First, on the basis of fixing R, solve the translation vector t. At this time, the objective function is:

[0127]

[0128] Based on the partial derivative of the current objective function, we can get: in, is the centroid of the weighted average, It can be written as:

[0129] Secondly, after obtaining t, solve for R and obtain:

[0130] Derive the formula: because, is a scalar, according to the scalar property a=a T , we can get: Substitute and solve for R:

[0131] According to the properties of matrix trace, we know that:

[0132] Define the covariance matrix S = XWY T , perform SVD decomposition on S, and get: S = U∑V T , we can get: tr(WY T RX)=tr(∑V T RU). According to the properties of orthogonal matrix and SVD decomposition, we can solve and calculate R = VU T .

[0133] Assume that the threshold of the iterative error is e. If E(R, t)>e, then a new Q' is obtained according to the transformation of R and t, and the corresponding point pairs are continued to be found. This iteration is continued until E(R, t) is less than the set threshold e.

[0134] Furthermore, this embodiment introduces the process of rendering based on the current body surface point cloud in step S05. The current body surface point cloud is directly rendered using multi-channel rendering + surface splashing + point sprite rendering technology to obtain a target image.

[0135] 1. The process of multi-channel rendering of the current body surface point cloud through three channels is as follows:

[0136] First rendering pass: Enable depth test, disable color writing, enable depth writing, disable blending, disable point sprites, adjust the depth of each fragment in the particle according to the particle radius, render the current surface point cloud, and output the depth texture map. In theory, each pixel on the screen must be calculated by the shader, and if there are objects in front and behind, overlaps, and blending, a pixel may be calculated by multiple fragment shaders;

[0137] Second rendering channel: turn off depth test, enable color writing, turn off depth writing, enable blending, set blending formula, use the depth value of each fragment in the first rendering channel to adjust the color value of each fragment, render the depth texture map, and output the color texture map;

[0138] Pass 3: Turn off blending, draw a full-screen quad in normalized device coordinates, and apply the color texture map from Pass 2.

[0139] 2. The process of rendering by surface splash technique:

[0140] The point-based object is represented as a set of non-connected points {P k}, which is similar to the radially symmetric basis function r k and represent the coefficients of the red, green and blue color components respectively Related, use w k As a generalized representation.

[0141] Select any point Q on the surface to establish a functional expression between Q and its surrounding minimal range {Pk}. k Considered as a two-dimensional plane with coordinates u and u k , define the function of this two-dimensional plane as the weighted sum of each point:

[0142] f c (u)=Σ k∈N w k r k (uu k ),

[0143] Given the mapping relationship from the function plane to the screen x = m(u): R 2 →R 2 , you need to go through the following three steps:

[0144] (1) f c (u) is transformed into a screen area to generate a continuous area signal:

[0145] g c (x)=(f c ☉m -1 )(x)=fc (m -1 (x)),

[0146] Among them, ⊙ represents the connection of the function.

[0147] (2) The band-limited screen area signal uses the initial filter h to calculate the continuous function:

[0148]

[0149] in, Represents a convolution operation.

[0150] (3) Sample the result of the continuous function and multiply the continuous result by the pulse sequence j(x) to obtain the discrete result: g(x) = g′ c (x)j(x).

[0151] (4) Reverse expansion of the above relationship can lead to a detailed expression:

[0152]

[0153] in, ρ k (x) is the resampling kernel. Independently transform and filter each basis function r k To construct the resampling kernel ρ k ,The process of accumulating and calculating these kernels in the screen area is called surface splashing.

[0154] In addition, ρ k (x) is further simplified by replacing m(u) with its k Local radiation estimation:

[0155]

[0156] Among them, x k =m(u k ), the Jacobian matrix Further calculations yield:

[0157]

[0158] in, represents the transformed basis function. Although the texture function is defined on an irregular grid, the resampling kernel ρ of the screen area k (x) can still be expressed as the transformed basis function r' k The form of convolution with the low-pass filter kernel h.

[0159] 3. Point sprite technology rendering process:

[0160] There is a built-in read-only variable gl_PointCoord in the fragment shader that interpolates the texture coordinates at the vertex.

[0161] There is a built-in read-only variable gl_PointSize in the vertex shader, which is used to control the final raster size of the point, which is a pixel value. The point size can be determined according to the point distance. The point distance transformation formula is:

[0162]

[0163] Among them, d is the distance from the point to the observation point, a, b, c are the parameters of the quadratic equation, which can be stored as a uniform value or set as a constant in the vertex shader. a controls the constant part of the final value, b controls the linear change of the final value with distance, and c controls the square relationship of the final value with distance transformation.

[0164] The following is a description of a contour scanning imaging device based on two modes provided in an embodiment of the present invention. The contour scanning imaging device based on two modes described below and the contour scanning imaging method based on two modes described above can be referred to each other.

[0165] First combine Figure 4 , the contour scanning imaging device based on two modes is introduced, such as Figure 4 As shown, the dual-modality contour scanning imaging device may include:

[0166] An image acquisition unit 100 is used to acquire a two-dimensional image and an infrared image of a human body to be tested;

[0167] The point cloud acquisition unit 200 is used to perform phase unwrapping on the two-dimensional image to obtain the current body surface point cloud;

[0168] The point cloud processing unit 300 is used to filter the special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on a preset coordinate conversion relationship, and determine the target area according to the infrared image;

[0169] The point cloud registration unit 400 is used to extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud;

[0170] The image rendering unit 500 is used to perform rendering based on the current body surface point cloud to obtain a target image.

[0171] The dual-modality contour scanning imaging apparatus provided in the embodiment of the present invention can be applied to dual-modality contour scanning imaging equipment. Figure 5 The hardware structure diagram of the dual-mode contour scanning imaging device is shown. Figure 5, the hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0172] In the embodiment of the present invention, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0173] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0174] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0175] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to implement each processing flow in the aforementioned contour scanning imaging solution based on two modalities.

[0176] An embodiment of the present invention further provides a storage medium, which can store a program suitable for execution by a processor, wherein the program is used to implement each processing flow in the aforementioned contour scanning imaging solution based on two modalities.

[0177] Finally, it should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. In this specification, various embodiments are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0178] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A contour scanning imaging method based on two modes, characterized in that: Methods include: Acquire a two-dimensional image and an infrared image of the human body to be tested; Perform phase unwrapping on the two-dimensional image to obtain the current body surface point cloud; Filter the special points in the current body surface point cloud, and project the current body surface point cloud onto the infrared image based on a preset coordinate transformation relationship, and determine the target area according to the infrared image; Extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud; Rendering is performed based on the current body surface point cloud to obtain the target image.

2. The dual-modality contour scanning imaging method according to claim 1, characterized in that: The process of rendering based on the current body surface point cloud to obtain the target image includes: Use multi-channel rendering + surface splash + point sprite rendering technology to render the current surface point cloud.

3. The dual-modality contour scanning imaging method according to claim 2, characterized in that: The process of rendering the current body surface point cloud using multi-channel rendering includes: Perform multi-channel rendering of the current body surface point cloud through three rendering channels; Among them, the rendering process of the three rendering channels is as follows: First rendering pass: Enable depth test, disable color writing, enable depth writing, disable blending, disable point sprites, adjust the depth of each fragment in the particle according to the particle radius, render the current surface point cloud, and output the depth texture map; Second rendering channel: turn off depth test, turn on point sprite, enable color writing, turn off depth writing, enable blending, set blending formula, use the depth value of each fragment in the first rendering channel to adjust the color value of each fragment, render the depth texture map, and output the color texture map; Pass 3: Turn off blending, draw a full-screen quad in normalized device coordinates, and apply the color texture map from pass 2.

4. The dual-modality contour scanning imaging method according to claim 1, characterized in that: The two-dimensional image is phase-unwrapped by the four-step phase shift method and the multi-frequency heterodyne method. The process includes: A four-step phase shift method is used to obtain each phase to be modulated of a two-dimensional image; Based on the phase to be modulated, the main phase value is obtained by solving the inverse tangent function; Through the multi-frequency heterodyne method, a unique principal value is obtained between [0,2π), and the phase difference of sinusoidal gratings of different frequencies is calculated to convert the high-frequency phase into a low-frequency phase so that the phase difference signal covers the entire field of view. Then, the absolute phase distribution of the two-dimensional image is obtained based on the phase difference.

5. The dual-modality contour scanning imaging method according to claim 1, characterized in that: Based on the preset coordinate transformation relationship, the process of projecting the current body surface point cloud onto the infrared image includes: According to the conversion relationship between the point cloud coordinate system and the infrared camera coordinate system, the coordinates of each point in the current body surface point cloud are converted from the point cloud coordinate system to the infrared camera coordinate system to obtain the infrared camera coordinates of each point, where the origin of the infrared camera coordinate system is the optical center of the lens, the x and y axes are parallel to the two sides of the imaging plane, and the z axis is the optical axis of the lens; According to the conversion relationship between the infrared camera coordinate system and the infrared image coordinate system, the infrared camera coordinates of each point are converted from the infrared camera coordinate system to the infrared image coordinate system to obtain the infrared image coordinates of each point, where the origin of the infrared image coordinate system is the midpoint of the imaging plane, and the x and y axes are parallel to the two sides of the imaging plane, respectively.

6. The dual-modality contour scanning imaging method according to claim 5, characterized in that: The process of converting the coordinates of each point in the current body surface point cloud from the point cloud coordinate system to the infrared camera coordinate system includes: The coordinates of each point in the current body surface point cloud are converted from the point cloud coordinate system to the infrared camera coordinate system through the formula. The formula is as follows: [x c y c z c ] T =R[x w y w z w ] T +t, Among them, x w ,y w ,z w They are the three coordinate components of point P in the point cloud coordinate system w, x c ,y c ,z c are the three coordinate components of point P in the infrared camera coordinate system c, R and t are two transformation matrices; The process of converting the infrared camera coordinates of each point from the infrared camera coordinate system to the infrared image coordinate system includes: Based on the principle of similar triangles, the infrared camera coordinates of each point are converted from the point cloud coordinate system to the infrared camera coordinate system through the formula. The formula is as follows:

7. The dual-modality contour scanning imaging method according to any one of claims 1 to 6, characterized in that: After extracting the body surface point cloud in the target area for point cloud registration and calculating the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud, the following steps are also included: Registering the current body surface point cloud with the reference body surface point cloud to obtain a single-modality registration result; The target area point cloud registration result and the single modality registration result are weighted averaged to obtain the target registration result.

8. A contour scanning imaging device based on two modes, characterized in that: Also includes: An image acquisition unit, used to acquire a two-dimensional image and an infrared image of a human body to be tested; A point cloud acquisition unit is used to perform phase unwrapping on the two-dimensional image to obtain a current body surface point cloud; A point cloud processing unit is used to filter special points in the current body surface point cloud, and based on a preset coordinate conversion relationship, project the current body surface point cloud onto an infrared image, and determine a target area according to the infrared image; A point cloud registration unit is used to extract the body surface point cloud in the target area for point cloud registration, calculate the difference between the current body surface point cloud and the reference body surface point cloud, and obtain the rotation and translation relationship between the current body surface point cloud and the reference body surface point cloud; The image rendering unit is used to render based on the current body surface point cloud to obtain a target image.

9. A contour scanning imaging device based on two modes, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the dual-modality based contour scanning imaging method as claimed in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the dual-modality contour scanning imaging method as described in any one of claims 1 to 7 is implemented.

Images