Method and system for creating a three-dimensional model based on real-time depth data

Abstract

The application discloses a method and system for creating a three-dimensional model based on real-time depth data, comprising the following steps: acquiring real-time RGB color images and depth images at three angles, obtaining RGB-depth image data at each angle according to the matching pixel space of the depth camera for sampling; performing exposure depth pixel speculation on the depth image at each angle, calculating the image pixel reliability Reliable, and combining the image pixel reliability Reliable with the RGB-depth image data to generate RGB-depth-reliable image data at three different angles; generating point clouds by hybrid calculation based on the depth camera parameters corresponding to the RGB-depth-reliable image data at three different angles, and generating a three-dimensional data model based on the point clouds; the method for creating a three-dimensional model can realize relatively clear and accurate creation of a three-dimensional model, and realizes real-time mapping relationship between the real world and the virtual world.

Description

Technical Field

[0001] This invention relates to the field of model creation technology, and in particular to a method and system for creating 3D models based on real-time depth data. Background Technology

[0002] Mixed Reality (MR) technology is a further development of virtual reality technology. This technology enhances the realism of the user experience by introducing real-world scene information into the virtual environment and establishing an interactive feedback loop between the virtual world, the real world, and the user.

[0003] However, the existing data mapping accuracy between the real world and the virtual world differs, resulting in a somewhat blurry and inconsistent 3D model that affects the user experience. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, this invention proposes a method and system for creating three-dimensional models based on real-time depth data, which can achieve the creation of three-dimensional models with relatively clear and accurate results, and realize the real-time data mapping relationship between the real world and the virtual world.

[0005] The present invention proposes a method for creating a three-dimensional model based on real-time depth data, which acquires RGB color images and depth images from three angles in real time, and obtains RGB-Depth image data for each angle based on the matching pixel space of the depth camera sampled from the two.

[0006] Exposure depth pixels are inferred for the depth map at each angle, and the image pixel reliability is calculated. The image pixel reliability is then merged with the RGB-Depth image data to generate RGB-Depth-Reliable image data at three different angles.

[0007] Based on the depth camera parameters corresponding to RGB-Depth-Reliable image data from three different angles, point clouds are generated through hybrid calculation, and a 3D data model is generated based on the point clouds.

[0008] Furthermore, in the process of acquiring RGB color images and depth maps from three angles in real time, and obtaining RGB-Depth image data for each angle based on the matching pixel space of the depth camera sampled from both, the merging process of the RGB-Depth image data is as follows:

[0009] Based on the physical parameters of the color camera and depth camera set on the same device, the correspondence between each pixel of the images captured by the color camera and depth camera is matched respectively, and then merged pixel by pixel.

[0010] Furthermore, the exposure depth pixel estimation for the depth map at each angle specifically includes:

[0011] The depth data of the depth map is optimized using a local adaptive image enhancement optimization algorithm;

[0012] Determine whether the pixel depth data of the Depth map is greater than the depth threshold;

[0013] If so, discard the image and re-acquire the depth map for that part;

[0014] If not, then perform a closing operation and median blurring on set A to obtain the filtered Depth depth map. Based on the average value of the range grid in the Depth depth map for each pixel neighborhood, obtain the image pixel reliability, where A(x,y)=1, and x and y are the pixel coordinates.

[0015] Furthermore, the optimization of depth data in the depth map using a local adaptive image enhancement optimization algorithm specifically includes:

[0016] Perform median averaging on the depth data DV(x,y) in the depth map;

[0017] Calculate the gradient map T(x,y) of the depth data DV(x,y) using the Sobel operator;

[0018] Using the Laplacian operator, find candidate points in depth data T(x,y) that have a local maximum threshold, and take the pixel coordinates of the candidate points;

[0019] Replace the candidate point values ​​in the depth data DV(x,y) with the depth values ​​of the Depth map obtained in step S1 to obtain the corrected depth data DV′(x,y);

[0020] Interpolate depth points into the depth data DV′(x,y) to obtain the pixel depth data of the optimized Depth map.

[0021] Furthermore, based on the depth camera parameters corresponding to RGB-Depth-Reliable image data from three different angles, point clouds are generated through hybrid calculation. The process of generating a 3D data model from these point clouds specifically includes:

[0022] Three sets of 3D coordinate data are calculated based on the physical spatial locations of the three depth cameras and the RGB-Depth-Reliable image data. The three sets of 3D coordinate data are then merged to obtain a complete 3D coordinate set.

[0023] The three-dimensional coordinate set is projected onto a plane from the four directions of front, back, left, and right to obtain four projection images;

[0024] The mapping relationship between the four projection images and the three-dimensional coordinate set is marked. The internal and external sets of the four sets of three-dimensional points are marked by the clustering algorithm. All external sets are merged to obtain the set of all three-dimensional external points.

[0025] Triangulation is performed on the set of three-dimensional external points to obtain a set of vertex index data. The set of vertex index data and the set of three-dimensional external points are used to generate three-dimensional model data.

[0026] A 3D model is obtained by interpolating and optimizing the 3D model data.

[0027] Furthermore, the RGB color images and depth maps from the three angles are obtained in the following way: three depth cameras are set up at 120 degrees, and each depth camera obtains the RGB color image and depth map in real time.

[0028] A system for creating 3D models based on real-time depth data includes an image data acquisition module, a model data generation module, and a data output module.

[0029] The image data acquisition module is used to acquire RGB color images and depth maps from three angles in real time. Based on the matching pixel space of the depth camera sampled from both, RGB-Depth image data for each angle is obtained.

[0030] The model data generation module is used to infer the exposure depth pixels of the Depth map at each angle, calculate the image pixel reliability, and merge the image pixel reliability with the RGB-Depth image data to generate RGB-Depth-Reliable image data at three different angles.

[0031] The data output module is used to obtain RGB-Depth-Reliable image data from three different angles, and to generate point clouds by mixing and calculating based on the depth camera parameters used in the three images, and then to generate a 3D data model based on the point clouds.

[0032] Furthermore, the model data generation module includes an image optimization module, a judgment module, a reset module, and a reliability output module;

[0033] The image optimization module is used to optimize the depth data of the depth map using a local adaptive image enhancement optimization algorithm;

[0034] The judgment module is used to determine whether the pixel depth data of the Depth map is greater than the depth threshold. If yes, it enters the reset module; otherwise, it enters the reliability output module.

[0035] The reset module is used to discard the image and reacquire the depth map of that part;

[0036] The reliability output module is used to perform closing operations and median fuzzing on set A to obtain the filtered Depth depth map. The image pixel reliability is obtained based on the average value of the range grid in the Depth depth map for each pixel neighborhood; where A(x,y)=1, and x and y are the pixel coordinates respectively.

[0037] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor and executed as described above in the method for creating a three-dimensional model.

[0038] The advantages of the method and system for creating 3D models based on real-time depth data provided by this invention are as follows: The method and system for creating 3D models based on real-time depth data provided in this invention can achieve the creation of 3D models with relatively clear and accurate results, realize the real-time data mapping relationship between the real world and the virtual world, and improve the user experience. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of the present invention; Detailed Implementation

[0040] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0041] like Figure 1 As shown, the method for creating a 3D model based on real-time depth data proposed in this invention includes the following steps:

[0042] S1: Real-time acquisition of RGB color images and depth images from three angles, and obtaining RGB-Depth image data for each angle based on the matching pixel space of the depth camera sampled from both.

[0043] Two cameras are set on the same device: a color camera and a depth camera. The color camera is used to acquire RGB color images, and the depth camera is used to acquire depth images. The matching pixel space of the color camera and the depth camera is known.

[0044] Three devices are set up, each equipped with a color camera and a depth camera. The three devices are arranged at 120 degrees. The two cameras on each device acquire RGB color images and depth images in real time, resulting in three sets of RGB color images and three sets of depth images. The RGB color images and depth images of each set are merged according to the matching pixel space of the corresponding cameras. Specifically, the physical parameters of the color camera and depth camera set on the same device are used to match the pixel correspondence of each image captured by the color camera and depth camera, and then the images are merged pixel by pixel.

[0045] S2: Perform exposure depth pixel estimation on the depth map at each angle, calculate the image pixel reliability, and merge the image pixel reliability with the RGB-Depth image data to generate RGB-Depth-Reliable image data at three different angles; specifically including steps S21 to S24:

[0046] S21: Optimize the pixel depth data of the Depth map using a local adaptive image enhancement optimization algorithm;

[0047] Local adaptive image enhancement is used to optimize the depth map, which addresses defects such as unclear image details, uneven brightness distribution due to uneven illumination, and noise contamination of the original image during imaging. The specific optimization process is as follows: steps S211 to S215.

[0048] S211: Perform median averaging on the depth data DV(x,y) in the depth map;

[0049] S212: Calculate the gradient map T(x,y) of depth data DV(x,y) using the Sobel operator;

[0050] S213: Using the Laplacian operator, find candidate points in depth data T(x,y) that have a local maximum threshold, and take the pixel coordinates of the candidate points;

[0051] S214: Replace the candidate point values ​​in the depth data DV(x,y) with the depth values ​​of the Depth map obtained in step S1 to obtain the corrected depth data DV′(x,y);

[0052] S215: Perform interpolation depth point processing on the depth data DV′(x,y) to obtain the pixel depth data of the optimized Depth depth map.

[0053] The formula for interpolation depth point processing is expressed as follows:

[0054] limn→∞Pn(x,y)=Pn-1(x,y)+Rn(x,y) / 4

[0055] Where P(x,y) represents the pixel depth data of the optimized Depth map, and R(x,y) is the Laplacian operator, where any point R(x,y) = 0 is chosen to make the surface smooth.

[0056] Calculate the P-value pixel by pixel.

[0057] Through steps S211 to S215, the depth map is optimized, resulting in a smooth depth map.

[0058] S22: Determine whether the pixel depth data of the Depth map is greater than the depth threshold. If yes, proceed to step S23; otherwise, proceed to step S24.

[0059] Whether each pixel meets the depth threshold requirement is one of the important conditions for the reliability of each pixel, and also an important condition for the reliability of image pixels in forming the Depth map.

[0060] S23: Discard the image and re-enter step S1;

[0061] S24: Perform a closing operation and median blurring on set A to obtain the filtered Depth depth map. Based on the average value of the range grid in the Depth depth map for each pixel neighborhood, obtain the image pixel reliability, where A(x,y)=1, and x and y are the pixel coordinates respectively.

[0062] S25: Image pixel reliability and RGB-Depth image data are merged to generate RGB-Depth-Reliable image data from three different angles.

[0063] Through steps S21 to S24, since the depth results of the depth camera are less reliable for pixels with excessive brightness, images with excessive brightness in the RGB color image and images with excessive depth in the Depth image are removed to obtain an image with higher clarity. Then, the RGB color image is closed and blurred by median, and mapped to the corresponding Depth image to calculate the image pixel reliability. Finally, a stable RGB-Depth-Reliable image is obtained, which is convenient for subsequent 3D projection and modeling processing based on the image.

[0064] S3: Based on the depth camera parameters corresponding to RGB-Depth-Reliable image data from three different angles, a point cloud is generated through hybrid calculation. A 3D data model is then generated based on the point cloud, specifically including steps S31 to S35:

[0065] S31: Calculate three sets of 3D coordinate data based on the physical spatial positions of the three depth cameras and the RGB-Depth-Reliable image data, and merge the three sets of coordinate data to obtain a complete 3D coordinate set.

[0066] Merging physical spatial location with RGB-Depth-Reliable image data confirms that all data on the same pixel are three-dimensional data, thus ensuring the integrity of the data on each pixel.

[0067] A three-dimensional spatial position is derived from the coordinates (xy) and depth of a pixel on an image (the coordinates and depth are superimposed to form three-dimensional data). These three data describe a three-dimensional spatial point, which is then transformed by a matrix.

[0068] S32: Project the three-dimensional coordinate set onto the plane from the four directions of front, back, left, and right to obtain four projection images;

[0069] S33: Mark the mapping relationship between the four projection images and the three-dimensional coordinate set, and mark the internal and external sets of the four sets of three-dimensional points respectively through clustering algorithm. Merge all external sets to obtain all three-dimensional external point sets.

[0070] The internal set is the part that can be optimized away, and the internal set is not visible in the final drawn interface.

[0071] S34: Triangulate the set of three-dimensional external points to obtain a set of vertex index data. The set of vertex index data and the set of three-dimensional external points are used to generate three-dimensional model data.

[0072] S35: Obtain a 3D model by interpolating and optimizing the 3D model data.

[0073] Through steps S31 to S35, a three-dimensional model is obtained based on the three-dimensional coordinate set of three angles and the corresponding projection map. When the image acquired by the depth camera changes, another set of three-dimensional models can be obtained through the above steps S1 to S3. Therefore, this embodiment can realize the output of dynamic three-dimensional models.

[0074] Steps S31 to S35 can be implemented in the following ways:

[0075] The three depth maps (x, y, depth) in the three RGB-Depth-Reliable image datasets are multiplied by their respective depth camera matrices to obtain three sets of three-dimensional coordinate data.

[0076] Merge the three sets of 3D coordinate data into one 3D coordinate set;

[0077] Calculate the boundary of the three-dimensional coordinate set, and define m, n, l small squares to represent the three-dimensional space;

[0078] Project the three-dimensional coordinate set onto the set small squares;

[0079] Remove the 3D coordinates from the set of 3D coordinates from the squares in the neighborhood where all the small squares are empty;

[0080] Based on the image pixel reliability (Reliable), further optimize and remove invalid 3D coordinates;

[0081] The final output is a set of filtered 3D coordinates. 3D model data is generated from this set of coordinates, and then the 3D model data is optimized by interpolation to obtain the 3D model.

[0082] Through the above steps S1 to S3, a relatively clear and accurate 3D model can be created, realizing the real-time data mapping relationship between the real world and the virtual world.

[0083] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for creating a three-dimensional model based on real-time depth data, comprising the following steps: S1, acquiring real-time RGB color images and Depth images at three angles, obtaining RGB-Depth image data at each angle according to the matching pixel space of the depth camera sampling the two; S2, performing exposure depth pixel speculation on the Depth image at each angle, calculating the image pixel reliability Reliable, and merging the image pixel reliability Reliable and the RGB-Depth image data to generate RGB-Depth-Reliable image data at three different angles; S3, based on the depth camera parameters corresponding to the RGB-Depth-Reliable image data at three different angles, hybrid calculation to generate point cloud, based on the point cloud to generate three-dimensional data model; wherein In the exposure depth pixel speculation of the Depth image at each angle, specifically comprising: S21, optimizing the depth data of the Depth image through a local adaptive image enhancement optimization algorithm; S22, judging whether the pixel depth data of the Depth image is greater than a depth threshold, if yes, entering step S23, if not, entering step S24; S23, discarding the image and re-entering step S1; S24, performing a closing operation and a median blur on set A to obtain a filtered Depth image, and obtaining the image pixel reliability Reliable based on the average value of each pixel neighborhood in the Depth image within a specified range, wherein A(x, y) = 1, x and y are pixel coordinates.

2. The method of claim 1, wherein, In the real-time acquisition of RGB color images and Depth images at three angles, the RGB-Depth image data is obtained according to the matching pixel space of the depth camera sampling the two, and the merging process of the RGB-Depth image data is as follows: According to the corresponding relationship between each pixel of the images captured by the matching color camera and depth camera respectively corresponding to the physical parameters of the color camera and depth camera set on the same device, and merging pixel by pixel.

3. The method of claim 1, wherein, In the optimization of the depth data of the Depth image through the local adaptive image enhancement optimization algorithm, specifically comprising: performing median average processing on the depth data DV(x, y) in the Depth image; find the gradient graph T(x, y) of the depth data DV(x, y) through the sobel operator; find the candidate point of the depth data T(x, y) having a local data maximum threshold through the Laplacian operator, and take the pixel coordinates of the candidate point; replace the candidate point value in the depth data DV(x, y) with the depth value of the Depth image obtained in step S1 to obtain the corrected depth data DV'(x, y); performing interpolation depth point processing on the depth data DV'(x, y) to obtain the pixel depth data of the optimized Depth image.

4. The method of claim 1, wherein, In the hybrid calculation to generate point cloud based on the depth camera parameters corresponding to the RGB-Depth-Reliable image data at three different angles, based on the point cloud to generate three-dimensional data model, specifically comprising: According to the physical space positions of the three depth cameras and the RGB-Depth-Reliable image data, three sets of three-dimensional coordinate set data are calculated, and the three sets of three-dimensional coordinate set data are combined to obtain a complete three-dimensional coordinate set; The three-dimensional coordinate set is projected from the front, back, left and right four directions to a plane to obtain four projection maps; The mapping relationship between the four projection maps and the three-dimensional coordinate set is marked, and four sets of internal and external three-dimensional point sets are marked by a clustering algorithm, respectively, and all external three-dimensional point sets are obtained by combining all external sets; Triangulation is performed on the three-dimensional external point set to obtain a vertex index data set, and a three-dimensional model data is generated from the vertex index data set and the three-dimensional external point set; The three-dimensional model data is optimized by interpolation to obtain a three-dimensional model.

5. The method of claim 1, wherein, The RGB color map and the Depth depth map of the three angles are obtained by the following method: three depth cameras are arranged at an angle of 120 degrees, and each depth camera obtains a real-time RGB color map and a Depth depth map.

6. A method system for creating a three-dimensional model based on real-time depth data, characterized by, It comprises an image data acquisition module, a model data generation module and a data output module; The image data acquisition module is used to obtain real-time RGB color maps and Depth depth maps of three angles, and to obtain RGB-Depth image data of each angle according to the matching pixel space of the depth cameras used for sampling; The model data generation module is used to perform exposure depth pixel speculation on the Depth depth map of each angle, to calculate image pixel reliability Reliable, and to combine the image pixel reliability Reliable with the RGB-Depth image data to generate RGB-Depth-Reliable image data of three different angles; The data output module is used to obtain RGB-Depth-Reliable image data of three different angles, respectively, to generate point clouds by mixed calculation based on the depth camera parameters used by the three, and to generate a three-dimensional data model based on the point clouds; The model data generation module comprises an image optimization module, a judgment module, a reset module and a reliability output module; The image optimization module is used to optimize the depth data of the Depth depth map by a local adaptive image enhancement optimization algorithm; The judgment module is used to judge whether the pixel depth data of the Depth depth map is greater than a depth threshold, if yes, the reset module is entered, and if not, the reliability output module is entered; The reset module is used to abandon the image and re-enter the image data acquisition module; The reliability output module is used to obtain a filtered Depth depth map by closed operation and median blur on set A, and to obtain image pixel reliability Reliable based on the average value of the range grid in the Depth depth map in each pixel field; wherein A(x,y)=1, x and y are pixel coordinates.

7. A computer readable storage medium, characterized in that, The computer readable storage medium stores a plurality of classification programs, which are used to be called and executed by the processor to create a three-dimensional model method according to any one of claims 1 to 5.

Images