Method for expanding and transmitting three-dimensional human point cloud based on bone points

Abstract

The application provides a three-dimensional human body point cloud unfolding and transmission method based on a skeleton point, and the implementation steps are as follows: color images and depth images of a human body under different perspectives are acquired; a sending end calculates three-dimensional skeleton point coordinates of the human body, sends the three-dimensional skeleton point coordinates, generates a human body color point cloud, divides the human body color point cloud, acquires a mixed data frame, and sends the mixed data frame; and a receiving end reconstructs the human body color point cloud. The sending end divides the human body color point cloud through the three-dimensional skeleton point of the human body, and forms a mixed data frame through the division result, so that background information and repeated information in multi-perspective human bodies can be removed, the defect that three-dimensional information sent in the prior art contains a large amount of redundant information is avoided, the bandwidth pressure of transmission can be reduced, and the transmission efficiency and the quality of the human body color point cloud reconstructed by the receiving end are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of point cloud processing, and relates to a three-dimensional human body point cloud unfolding and transmission method, in particular to a three-dimensional human body point cloud unfolding and transmission method based on skeleton points, which can be applied to the field of three-dimensional human body virtual social interaction. BACKGROUND

[0002] Three-dimensional human body point cloud is a collection of three-dimensional data representing the shape of a human body obtained through three-dimensional scanning technology, cameras, or other sensors. Each point in the three-dimensional human body point cloud has three-dimensional coordinates, usually , representing the spatial coordinates of different positions on the surface of the human body. These point cloud data can contain complete shape, posture, action, etc. of the human body and are widely used in many fields.

[0003] The basic idea of three-dimensional human body virtual social interaction is to obtain color images and depth images of a human body from multiple perspectives at the sending end through multiple cameras, encode and compress the image data, then remotely connect to the receiving end and transmit the data, and reconstruct the complete three-dimensional point cloud of the human body at the receiving end by decoding the received data.

[0004] For example, patent application CN115695441A, entitled "Three-dimensional human body virtual social interaction system and method based on P2P technology", discloses a method for transmitting three-dimensional human body data, which converts the collected color images, anti-noise encodes the depth images, and arranges the encoded color images and depth images to form a mixed data frame. This method can transmit all the data required for complete three-dimensional reconstruction of the human body, but because the transmitted color images and depth images contain a lot of redundant information such as background and repeated perspectives of the human body, the required bandwidth is large, which affects the speed and quality of transmitting three-dimensional data. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a three-dimensional human body point cloud unfolding and transmission method based on skeleton points to solve the technical problem of large required bandwidth caused by the transmission of a large amount of redundant information.

[0006] To achieve the above purpose, the technical solution adopted by the present application includes the following steps:

[0007] (1) Obtain color images and depth images of a human body from different perspectives;

[0008] through multiple Camera acquisition color image and depth image under a view angle, wherein ;

[0009] (2) the sending end calculates the three-dimensional skeleton point coordinates of the human body and sends them;

[0010] The color image captured by at least two The first two-dimensional skeleton point coordinates of the human body are calculated, and the human body skeleton point coordinate information is sent to the receiving end, wherein ;

[0011] (3) the sending end generates human body color point cloud;

[0012] Each depth image is back-projected, and the textureless point cloud obtained by each color image corresponding to the back-projection is colored to obtain human body color point cloud corresponding to each depth image;

[0013] (4) the sending end divides the human body color point cloud;

[0014] Each two human body three-dimensional skeleton points are connected into a skeleton segment, and after calculating the distance between each point in the human body color point cloud and each skeleton segment, the point is divided into the nearest skeleton segment to obtain multiple groups of human body color point clouds with the same number of skeleton segments;

[0015] (5) the sending end obtains a mixed data frame and sends it;

[0016] The position coordinate information and color information of each point in each group of human body color point cloud are respectively mapped into a distance grid and a color grid, and all the distance images and color images formed by the mapping are spliced respectively, and the mixed data frame formed by the splicing is sent to the receiving end;

[0017] (6) the receiving end obtains a three-dimensional human body point cloud transmission result;

[0018] The receiving end reconstructs the human body color point cloud through the human body skeleton point coordinate information sent in step (2) and the mixed data frame sent in step (5).

[0019] Compared with the prior art, the present application has the following advantages:

[0020] ​The sending end of the present application divides the human body color point cloud through human body three-dimensional skeleton points, and forms a mixed data frame through the division result, which can remove background information and repeated information in multi-view of the human body, avoids the defect that the existing technology contains a large amount of redundant information in sending three-dimensional information, can reduce the bandwidth pressure of transmission, and further improves the transmission efficiency and the quality of the reconstructed human body color point cloud of the receiving end. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The flowchart for the implementation of the present application. DETAILED DESCRIPTION

[0022] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0023] Reference Figure 1 , the present application comprises the following steps:

[0024] Step 1) obtaining color images and depth images of the human body under different views;

[0025] By the camera obtains color images and depth images under one view, wherein ;

[0026] Since the field of view of each camera is limited, if you want to capture a complete human body data, multiple cameras are needed. In the present embodiment, , four cameras are respectively used to capture color images and depth images of the front, left, right and back of the human body. Step 2) the sending end calculates the three-dimensional skeleton point coordinates of the human body and sends them;

[0027] The two-dimensional skeleton point coordinates of the first person in the color images captured by at least two cameras

[0028] are calculated to obtain the three-dimensional skeleton point coordinates of the human body, and the human body skeleton point coordinate information is sent to the receiving end, wherein ; The formula for calculating the three-dimensional skeleton point coordinates of the human body is:

[0029] ;

[0030] ;

[0031] , ;

[0032] wherein, is the two-dimensional skeleton point coordinates of the first person in the color images captured by at least two cameras ​​​​The intrinsic parameter matrix of a Taiwanese RGB-D camera. No. From the first RGB-D camera camera external parameters, Represents a rotation matrix. Represents the translation vector. For the first The three-dimensional coordinates of each skeletal point.

[0033] In this embodiment, by The human pose estimation library identifies the coordinates of 2D human skeleton points in color images of the front, left, and right sides, respectively. 3D skeleton point coordinates are then calculated using the skeleton point recognition results from the front and left, and front and right color images. The skeleton point coordinate information is transmitted using the text channel of WebRTC.

[0034] Step 3) The sending end generates a colored point cloud of the human body;

[0035] Backprojection is performed on each depth image, and the textureless point cloud obtained from the backprojection of each color image is then colored to obtain... Human body color point cloud corresponding to a depth image;

[0036] The coordinates of each depth image are Value Calculate the first unit corresponding to the pixel. Spatial points in the camera coordinate system ; and by using the coordinates in the color image corresponding to each depth image as pixels Value for this spatial point Coloring is performed, where:

[0037] ;

[0038] in, express The inverse result of .

[0039] Step 4) The sending end divides the human body color point cloud;

[0040] Connect every two human 3D skeletal points to form a skeletal segment, and calculate the distance between each point in the human color point cloud and each skeletal segment. Then assign the point to the nearest skeletal segment to obtain multiple sets of human color point clouds with the same number of skeletal segments.

[0041] The formula for calculating the distance from each point to each bone segment in a human body color point cloud is as follows:

[0042] ;

[0043] ;

[0044] ;

[0045] wherein is a projection scale factor, represents the coordinate of any point in the point cloud, and respectively are the coordinates of the two human three-dimensional skeletal points in the corresponding skeletal segment. In the embodiment, 15 groups of divided human color point clouds are obtained, representing different body parts of the human body.

[0046] Step 5) The sending end acquires the mixed data frame and sends;

[0047] The position coordinate information and color information of each point in each group of human color point clouds are respectively mapped into the distance grid and the color grid, and all the formed distance images and color images are spliced respectively, and then the spliced mixed data frame is sent to the receiving end;

[0048] (5a) Taking the center of the skeletal segment as the spherical center , the spherical coordinates of the point with the position coordinate in the color point cloud belonging to the skeletal segment about the center of the skeletal segment are calculated:

[0049] ;

[0050] wherein is the Euclidean distance from the point to the spherical center, is the azimuth angle formed by the point with the coordinate and the positive direction of the axis in the horizontal plane, and is the elevation angle formed by the line connecting the point with the coordinate and the spherical center and the positive direction of the axis;

[0051] (5b) The position coordinate information and color information of each point in each group of human color point clouds are respectively mapped into the two-dimensional distance image and color image with the width and height of . In the embodiment, the value range of the azimuth angle and the elevation angle of the point is , , the value of which respectively corresponds to the horizontal coordinate and the vertical coordinate in the image, and the pixel value corresponds to the distance from the point to the spherical center or the color information. All the obtained distance images and color images are spliced to form a mixed data frame. In the embodiment, the width and height of the mixed data frame are .

[0052] (5c) In this embodiment, the mixed data frame is sent to the receiving end through the video channel of WebRTC.

[0053] Step 6) The receiving end obtains the three-dimensional human point cloud transmission result;

[0054] (6a) The receiving end splits the mixed data frame sent by the sending end through the video channel to obtain the distance image and the color image; at the same time, the bone point coordinate information sent by the sending end through the text channel is connected according to the same connection mode as the sending end to obtain the same bone segment as the sending end;

[0055] (6c) Calculate the center of the bone segment The coordinates of the three-dimensional point corresponding to all pixels in the corresponding distance image , obtain the human color point cloud corresponding to the bone segment, wherein:

[0056] ;

[0057] Wherein, , respectively, the horizontal coordinate and the vertical coordinate of the pixel point of the depth image;

[0058] (6d) Through the value of the pixel point in the color image whose pixel coordinate is the same as the pixel coordinate of the depth image , the corresponding three-dimensional point is colored, realizing the reconstruction of the human color point cloud.

Claims

1. A method for skeleton point-based three-dimensional human point cloud unfolding and transmission, characterized in that, Includes the following steps: (1) Obtain color and depth images of the human body from different perspectives; By station The camera acquires color images and depth images at individual perspectives, wherein ; (2) The sending end calculates and sends the coordinates of the three-dimensional skeleton points of the human body; by at least two the first personal two-dimensional skeleton point coordinates calculate its three-dimensional skeleton point coordinates, and send the human skeleton point coordinate information to the receiving end, wherein ; (3) The sending end generates a human body color point cloud; For each depth image, a back-projection is performed and the resulting untextured point cloud is colored by each color image corresponding to the depth image, resulting in a human color point cloud corresponding to the depth image; (4) The transmitting end divides the human body color point cloud; Connect every two human 3D skeletal points to form a skeletal segment, and calculate the distance between each point in the human color point cloud and each skeletal segment. Then assign the point to the nearest skeletal segment to obtain multiple sets of human color point clouds with the same number of skeletal segments. (5) The sending end acquires the mixed data frame and sends it; The position coordinates and color information of each point in each group of human body color point clouds are mapped to the distance grid and color grid respectively. The distance images and color images formed by all mappings are stitched together, and the stitched mixed data frame is sent to the receiving end. (6) The receiving end acquires the transmission results of the three-dimensional human body point cloud; The receiving end reconstructs the human body color point cloud using the human skeleton point coordinate information sent in step (2) and the mixed data frame sent in step (5).

2. The method of claim 1, wherein, The formula for calculating the coordinates of the three-dimensional skeleton points of the human body in step (2) is as follows: ； ， ； in, For the first The intrinsic parameter matrix of a Taiwanese RGB-D camera. No. From the first RGB-D camera camera external parameters, Represents a rotation matrix. Represents the translation vector. For the first The three-dimensional coordinates of each skeletal point.

3. The method of claim 2, wherein, The steps for generating the human body color point cloud in step (3) are as follows: by calculating for each pixel of each depth image its corresponding 1st camera coordinate of the space point where the coordinates of the pixel are with value and coloring the space point by the value of the pixel of the corresponding color image of each depth image with coordinates where the coordinates of the pixel are​ ； wherein represents the inverse of the result of the calculation of 4. The method of claim 1, wherein, The calculation formula for the distance from each point in the human body color point cloud to each bone segment, as described in step (4), is as follows: ； ； ； wherein is a projection scaling factor, represents a coordinate of an arbitrary point in the point cloud, and are respectively coordinates of two human three-dimensional skeletal points in the corresponding skeletal segment.

5. The method of claim 1, wherein, The steps for obtaining the mixed data frame in step (5) are as follows: (5a) taking the center of the bone segment as the sphere center , calculating the spherical coordinates of the points of the color point cloud belonging to the bone segment with position coordinates with respect to the center of the bone segment : ； in Let Euclidean distance be the distance from the point to the center of the ball. Coordinates are The point on the horizontal plane and The azimuth angle formed by the positive direction of the axis. Coordinates are The line connecting the point and the center of the ball and The angle of elevation formed in the positive direction of the axis; (5b) mapping the position coordinate information and color information of each point in each group of human color point clouds into a two-dimensional distance image and a color image with a width and height of respectively, and stitching all the obtained distance images and color images to form a mixed data frame.

6. The method according to claim 5, characterized in that, The steps for the receiving end to acquire the 3D human point cloud transmission result in step (6) are as follows: (6a) The receiver splits the mixed data frame sent by the transmitter through the video channel to obtain a distance image and a color image; at the same time, it connects the skeletal point coordinate information sent by the transmitter through the text channel in the same way as the transmitter to obtain the same skeletal segments as the transmitter. (6c) Calculation of the center of the skeletal segment The corresponding distance is to the coordinates of the three-dimensional points corresponding to all pixels in the image. The human body color point cloud corresponding to the skeletal segment is obtained, where: ； in, , These are the x and y coordinates of the pixels in the depth image, respectively. (6d) By using the pixels in the color image whose pixel coordinates are the same as those in the depth image The values ​​are used to color the corresponding 3D points, thereby reconstructing the human body's colored point cloud.

Images