Pose alignment method, three-dimensional reconstruction method, electronic equipment and storage medium

Abstract

The invention provides a pose alignment method, a three-dimensional reconstruction method, electronic equipment and a storage medium, and relates to the technical field of data processing, and the pose alignment method comprises the steps: obtaining RGBD data shot by a camera and internal parameters of the camera; the RGBD data comprises RGB data and depth data, and pixel points in the RGB data are in one-to-one correspondence with pixel points in the depth data; determining a first feature point corresponding to the pixel point according to the internal reference and the RGB data, and determining a second feature point corresponding to the pixel point according to the internal reference and the depth data; and calculating a pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system based on the pixel point, the first feature point and the second feature point so as to align the pose three-dimensional coordinate system with the depth map three-dimensional coordinate system. According to the method, the pose scale can be quickly, accurately and stably obtained, and the alignment process of the RGB data is optimized, so that the RGBD data can be better used.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a posture alignment method, a three-dimensional reconstruction method, an electronic device and a storage medium. Background Art

[0002] Pose alignment plays an important role in three-dimensional (3D) computer vision, and is particularly important in the fusion of multi-frame 3D data with differences between the temporal dimension and the spatial dimension. In the use of RGBD data, after obtaining the pose according to RGB (Red Green Blue) data, in order to better use the RGBD (Red Green Blue and Depth) data, it is necessary to determine the scale between the depth data and the pose. The existing solution manually adds markers to the pose 3D data, actually measures the real data of the object, and manually determines the real data corresponding to the markers in the measured data to obtain the scale between the depth data and the pose. This solution has poor stability, strong randomness, and poor noise resistance. Summary of the invention

[0003] The embodiments of the present application provide a posture alignment method, a three-dimensional reconstruction method, an electronic device and a storage medium to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a posture alignment method, comprising: acquiring RGBD data captured by a camera and an internal parameter of the camera; wherein the RGBD data includes RGB data and depth data, and the pixels in the RGB data and the depth data correspond one to one; determining a first feature point corresponding to the pixel point according to the internal parameter and the RGB data, and determining a second feature point corresponding to the pixel point according to the internal parameter and the depth data; wherein the first feature point is located in a posture three-dimensional coordinate system, and the second feature point is located in a depth coordinate system. Figure 3 3D coordinate system; based on the pixel point, the first feature point, and the second feature point, the pose 3D coordinate system and the depth Figure 3 The pose scale between the three-dimensional coordinate system to convert the pose three-dimensional coordinate system to the depth Figure 3 dimensional coordinate system alignment.

[0005] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction method, comprising: obtaining aligned pose data; wherein the aligned pose data is obtained according to the above method; and generating a three-dimensional reconstruction result based on the aligned pose data.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0008] Compared with the related art, this application has the following advantages:

[0009] According to an embodiment of the present application, RGBD data captured by a camera and an internal parameter of the camera are first obtained, and according to the one-to-one correspondence between the RGB data in the RGBD data and the pixel points in the depth data, a first feature point corresponding to the pixel point is determined based on the camera internal parameter and the RGB data, and a second feature point corresponding to the same pixel point is determined based on the internal parameter and the depth data, wherein the first feature point is located in the three-dimensional coordinate system of the pose, and the second feature point is located in the depth. Figure 3 The three-dimensional coordinate system is then used to calculate the position three-dimensional coordinate system and the depth based on the pixel point, the first feature point, and the second feature point. Figure 3 The pose scale between the three-dimensional coordinate system to convert the pose three-dimensional coordinate system to the depth Figure 3 The embodiment of the present application can quickly, accurately and stably obtain the pose scale, and optimize the alignment process of RGB data to better use RGBD data.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0012] Figure 1 A flow chart of a posture alignment method provided in an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of a scenario provided in an embodiment of the present application is shown;

[0014] Figure 3A flowchart of an implementation of a posture alignment method provided in an embodiment of the present application is shown;

[0015] Figure 4 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application is shown;

[0016] Figure 5 A structural block diagram of a position alignment device provided in an embodiment of the present application is shown;

[0017] Figure 6 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0019] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0020] First, the basic knowledge of mathematical projection transformation involved in this scheme is introduced.

[0021]

[0022]

[0023]

[0024] In the above formula, f x 、f y is the focal length of the camera, in pixels; u0 and v0 are the principal points in the x and y directions; f x 、f y , u0, v0 constitute the internal parameter matrix. c , Y c , Z c is a 3D point in the camera coordinate system, and u and v are pixel coordinates in the pixel coordinate system.

[0025] According to formula (1), the 3D point in the camera coordinate system can be mapped to the pixel coordinate system based on the camera intrinsic parameters.

[0026] From equations (2) and (3), the depth value and pixel coordinates are known. According to the camera intrinsic parameters, the depth map can be converted to the corresponding camera coordinate system.

[0027] In the existing pose alignment scheme, marker points are manually added to the 3D data, and the pose scale alignment is achieved by obtaining the conversion relationship through the marker points. The specific method includes: manually measuring the real value of the object captured by the image, and after obtaining the pose 3D point, establishing the conversion relationship between the pose coordinates and the real value. The scheme has a low degree of automation and requires human participation; it has poor stability, strong randomness, and poor noise resistance.

[0028] Based on this, the present application provides a posture alignment method, a three-dimensional reconstruction method, an electronic device and a storage medium. The method eliminates human participation and ensures full process automation. After actual verification, the present application can resist the influence of a certain degree of noise and has high stability.

[0029] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0030] This embodiment of the application provides a posture alignment method. Figure 1 The flowchart of a posture alignment method according to an embodiment of the present application is shown, and the method may include:

[0031] Step 101, obtaining RGBD data captured by a camera and an internal parameter of the camera; wherein the RGBD data includes RGB data and depth data, and pixels in the RGB data and the depth data correspond one to one.

[0032] In an embodiment of the present application, the same or different cameras can be used to shoot the same scene, and the scene can be selected according to actual needs, and the embodiment of the present application does not specifically limit this. For example, the scene may include one or more of people, animals, plants, buildings, scenery, indoor environments, etc. A camera is used to shoot multiple sets of depth maps (i.e., RGBD data), and each RGBD data and the internal parameters of the camera that shoots the RGBD data are recorded, that is, the RGBD data shot by the camera and the internal parameters of the camera are obtained. It should be noted that the RGBD data includes RGB data and depth data, wherein the depth data can be a depth value. For a pixel point in an image, its depth value represents the distance from the point in the three-dimensional world corresponding to this pixel point to the camera. The pixels in the RGB data and the depth data correspond one to one. For example, the RGB data includes three pixels: point A, point B, and point C, so the depth data also includes the depth data corresponding to the pixel points A, point B, and point C.

[0033] It should be noted that the camera in the embodiment of the present application may be an RGB-D camera system. In an RGB-D camera system, the pose data (also called posture data) of the same photo is closely related to the depth data.

[0034] In this step, the RGB data and depth data captured by the camera are obtained. The pixels in the RGB data and the depth data correspond one to one, so that the depth data and RGB data of the same pixel can be processed, and the relationship between the processed data can be established, providing a data basis for calculating the posture scale.

[0035] Step 102, determining a first feature point corresponding to the pixel point according to the intrinsic parameter and the RGB data, and determining a second feature point corresponding to the pixel point according to the intrinsic parameter and the depth data; wherein the first feature point is located in a three-dimensional coordinate system of position and posture, and the second feature point is located in a depth coordinate system of position and posture. Figure 3 dimensional coordinate system.

[0036] In the embodiments of the present application, the pose refers to the position and orientation of the camera in three-dimensional space. Specifically, the pose of the camera includes: position, which is the position of the camera in three-dimensional space, usually represented by a three-dimensional vector; direction, which is the orientation of the camera, which can be represented by a rotation matrix or equivalent quaternion, Euler angle, etc. The pose three-dimensional coordinate system is used to describe the position and orientation of the camera in three-dimensional space when taking a photo.

[0037] A depth map is an image used to describe the distance from each pixel in the scene to the camera. The depth map coordinate system is used to represent how the depth values ​​in the depth map are mapped to points in three-dimensional space.

[0038] A pixel point is a two-dimensional point in the pixel coordinate system corresponding to the camera. According to the internal parameter and the RGB data, the first feature point corresponding to the pixel point is determined, that is, the pixel point to be processed is determined according to the RGB data, and these pixel points are converted from two dimensions to three dimensions in combination with the internal parameter to obtain the first feature point corresponding to the pixel point. The first feature point is located in the three-dimensional coordinate system of the pose. According to the internal parameter and the depth data, the second feature point corresponding to the pixel point is determined, that is, the depth data corresponding to the pixel point is converted to three dimensions using the internal parameter to obtain the second feature point corresponding to the pixel point. The second feature point is located in the depth coordinate system. Figure 3 dimensional coordinate system.

[0039] In this step, based on the internal parameters, the three-dimensional points corresponding to the RGB data and the three-dimensional points corresponding to the depth data are calculated respectively, so as to obtain the three-dimensional coordinate system for representing the posture and the depth. Figure 3 The three-dimensional point of the two-dimensional coordinate system is obtained so as to analyze and calculate the first feature point and the second feature point, and then determine the transformation relationship between the two coordinate systems.

[0040] Step 103, calculating the three-dimensional coordinate system of the pose and the depth based on the pixel point, the first feature point, and the second feature point Figure 3 The pose scale between the three-dimensional coordinate system to convert the pose three-dimensional coordinate system to the depth Figure 3 dimensional coordinate system alignment.

[0041] In the embodiment of the present application, after obtaining the pixel point, the first feature point, and the second feature point, the relationship between the pose three-dimensional coordinate system and the depth is obtained by analyzing and calculating the relationship between the data. Figure 3 The pose scale is used to determine the pose three-dimensional coordinate system and the depth Figure 3 Therefore, the pose scale can be used to convert the pose three-dimensional coordinate system to the depth Figure 3 Then, the aligned RGB data is obtained for 3D reconstruction.

[0042] In this step, the three-dimensional coordinate system of the pose and the depth are obtained by analyzing and calculating the first feature point and the second feature point corresponding to the same pixel. Figure 3 The pose scale between the two coordinate systems can be used to align the point data in the two coordinate systems.

[0043] According to an embodiment of the present application, RGBD data captured by a camera and an internal parameter of the camera are first obtained, and according to the one-to-one correspondence between the RGB data in the RGBD data and the pixel points in the depth data, a first feature point corresponding to the pixel point is determined based on the camera internal parameter and the RGB data, and a second feature point corresponding to the same pixel point is determined based on the internal parameter and the depth data, wherein the first feature point is located in the three-dimensional coordinate system of the pose, and the second feature point is located in the depth. Figure 3 The three-dimensional coordinate system is then used to calculate the position three-dimensional coordinate system and the depth based on the pixel point, the first feature point, and the second feature point. Figure 3 The pose scale between the three-dimensional coordinate system to convert the pose three-dimensional coordinate system to the depth Figure 3 The embodiment of the present application can quickly, accurately and stably obtain the pose scale, and optimize the alignment process of RGB data to better use RGBD data.

[0044] In a possible implementation, the three-dimensional coordinate system of the pose and the depth are calculated based on the pixel point, the first feature point, and the second feature point. Figure 3The pose scale between the three-dimensional coordinate system can be performed according to the following steps: determine a plurality of pixels to be processed in the RGB data, determine a current pixel and a designated pixel in the plurality of pixels to be processed; wherein the designated pixel is any pixel in the plurality of pixels to be processed except the current pixel; calculate the spatial distance between the current pixel and the designated pixel based on the first feature point and the second feature point, and calculate the pose three-dimensional coordinate system and the depth according to the spatial distance. Figure 3 The pose scale between two-dimensional coordinate systems.

[0045] In this possible implementation, the pixel points to be processed are a part of key points selected from multiple pixel points of RGB data, for example, some pixel points with significant features and easy to detect and describe, and may include multiple points, namely point A, point B, point C and point D. A current pixel point and a designated pixel point are determined from the multiple pixel points to be processed, wherein the current pixel point is the point currently involved in the calculation, for example, point A, and the designated pixel point is any pixel point other than the current pixel point from the multiple pixel points to be processed, for example, any one of points B, C, and D may be selected as the designated pixel point.

[0046] The spatial distance between the current pixel and the designated pixel is calculated based on the first feature point and the second feature point. For example, if the current pixel is point A and the designated pixel is point B, the spatial distance between point A and point B is calculated. It should be noted that the pixel points other than the current pixel in the pixel points to be processed can be selected as designated pixel points in turn, that is, point C and point D can also be used as designated pixel points, and then the spatial distance is calculated in combination with the current pixel point, that is, after obtaining the spatial distance based on point A and point B, the spatial distance can also be obtained based on point A and point C, and the spatial distance can be obtained based on point A and point D. In addition, when point B is the current pixel, point C and point D can be determined as designated pixel points in turn, and the spatial distance can be obtained based on point B and point C, and the spatial distance can be obtained based on point B and point D.

[0047] After obtaining the spatial distance between the current pixel and the specified pixel, the three-dimensional coordinate system of the pose and the depth are calculated according to the spatial distance. Figure 3 The pose scale between two dimensional coordinate systems.

[0048] In this possible implementation, the pose scale is calculated by calculating the spatial distance between the current pixel and the designated pixel, so that the obtained pose scale is more accurate.

[0049] In a possible implementation, calculating the spatial distance between the current pixel and the designated pixel based on the first feature point and the second feature point can be performed according to the following steps: calculating the first spatial distance between the first feature point corresponding to the current pixel and the first feature point corresponding to the designated pixel, and calculating the second spatial distance between the second feature point corresponding to the current pixel and the second feature point corresponding to the designated pixel.

[0050] In this possible implementation, for example, when point A is the current pixel point and point B is the designated pixel point, the first spatial distance between the first feature point corresponding to the current pixel point and the first feature point corresponding to the designated pixel point is calculated, that is, the first spatial distance between the first feature point of point A and point B is calculated, and the second spatial distance between the second feature point corresponding to the current pixel point and the second feature point corresponding to the designated pixel point is calculated, that is, the second spatial distance between the second feature point of point A and point B is calculated. It should be noted that after completing the calculation of the spatial distance, that is, obtaining the first spatial distance and the second spatial distance, the calculation results need to be screened, that is, it is necessary to ensure that the first spatial distance is greater than 0, and the second spatial distance is greater than 0. In this possible implementation, the first spatial distance between the first feature point of the current pixel point and the designated pixel point, and the second spatial distance between the second feature point are calculated respectively, so as to establish the corresponding three-dimensional coordinate system of posture and depth respectively. Figure 3 The relationship between dimensional coordinate systems.

[0051] In a possible implementation, the position three-dimensional coordinate system and the depth are calculated according to the spatial distance. Figure 3 The posture scale between the three-dimensional coordinate system can be performed according to the following steps: calculating the ratio of the first spatial distance to the second spatial distance, and determining the posture three-dimensional coordinate system and the depth according to the ratio. Figure 3 The pose scale between two dimensional coordinate systems.

[0052] In this possible implementation, the distances calculated based on different designated pixel points form distance pairs, and the distance ratio in the distance pairs is calculated. For example, when the current pixel point is point A, for the case where point B is the designated pixel point, the first spatial distance and the second spatial distance between points A and B are calculated, and the ratio of the first spatial distance to the second spatial distance is calculated. For the case where point C is the designated pixel point, a distance ratio is calculated for points A and C. For the case where point D is the designated pixel point, a distance ratio can also be calculated for points A and D. The multiple ratios finally obtained are analyzed and processed to obtain the three-dimensional coordinate system of the pose and the depth. Figure 3 The pose scale between two dimensional coordinate systems.

[0053] In this possible implementation, by calculating the spatial distances between the two three-dimensional points corresponding to the current pixel point and the two three-dimensional points corresponding to the specified pixel point, a distance pair corresponding to the current pixel point is obtained, and then the ratio of the distances in the distance pair is obtained. The posture scale is determined based on the ratio to improve the accuracy of the posture scale.

[0054] In a possible implementation, the three-dimensional coordinate system of the pose and the depth are calculated based on the pixel point, the first feature point, and the second feature point. Figure 3 Before the posture scale between the two-dimensional coordinate systems, it also includes: sorting the first feature points corresponding to the RGB data to obtain a first point sequence, and sorting the second feature points corresponding to the RGB data according to the pixel point order corresponding to the first point sequence to obtain a second point sequence; determining the current pixel point among the multiple pixel points to be processed, including: determining the current pixel point among the multiple pixel points to be processed according to the pixel point order corresponding to the first point sequence.

[0055] In this possible implementation, the first feature points corresponding to the RGB data are sorted, and since each first feature point corresponds to a pixel point, the first point sequence is arranged in the order of the pixels. The second feature points corresponding to the pixels are sorted in the same order, that is, the second feature points corresponding to the RGB data are sorted in the order of the pixels corresponding to the first point sequence to obtain a second point sequence.

[0056] In this possible implementation, among the multiple pixel points to be processed, the current pixel point is determined according to the order of pixel points corresponding to the first point sequence, so as to obtain data required for the steps of calculating the first spatial distance between the first feature point corresponding to the current pixel point and the first feature point corresponding to the specified pixel point in the RGB data, and calculating the second spatial distance between the second feature point corresponding to the current pixel point and the second feature point corresponding to the specified pixel point according to the first point sequence and the second point sequence. The first point sequence and the second point sequence are obtained by sorting the feature points, thereby reducing the complexity of the data search process caused by random storage of data, improving data processing efficiency, and ensuring calculation accuracy.

[0057] In a possible implementation manner, the three-dimensional coordinate system of the posture and the depth are determined according to the ratio. Figure 3 The posture scale between the three-dimensional coordinate system can be performed according to the following steps: sorting the ratios to obtain a ratio sorting result; determining multiple target ratios in the ratio sorting result according to a preset ratio screening condition, and calculating the mean and standard deviation of the multiple target ratios; when the standard deviation is less than a preset standard deviation threshold, using the mean as the ratio between the posture three-dimensional coordinate system and the depth Figure 3The pose scale between two dimensional coordinate systems.

[0058] In this possible implementation, the preset ratio screening condition can be used to screen some or all of the ratios in the ratio sorting result to obtain the target ratio. The specific content of the screening condition can be set according to actual needs, and the embodiment of the present application is not specifically limited to this. For example, the ratio of the specified position can be screened in the order of arrangement to obtain the target ratio, or by setting the position range, the target ratio within the range can be obtained. The ratio screening condition can also be used to ensure the validity of the screening result, for example, the result of screening ratio greater than 0 obtains the target ratio.

[0059] In this possible implementation, by sorting and screening by comparison values, redundant and noisy data are removed, and the impact of noise on the posture scale is reduced. By setting the standard deviation threshold, a mean with better performance can be screened out to obtain a posture scale with greater stability.

[0060] In a possible implementation, based on preset ratio screening conditions, multiple target ratios are determined in the ratio sorting results, which can be performed according to the following steps: deduplicating the ratios in the ratio sorting results; taking the ratio between the first position and the second position in the ratio sorting results after deduplication as the target ratio; wherein the first position is before the second position.

[0061] In this possible implementation, by removing duplicates from the ratios in the ratio sorting results, and by setting the first position and the second position to screen the middle ratios in the ratio sorting results as the target ratios, the influence of noise can be reduced to a certain extent and the stability of the results can be improved. It should be noted that the first position and the second position can be set according to actual needs, and the embodiment of the present application does not specifically limit this. Preferably, the first position can be in the first 20% of the sorting results, and the second position can be in the last 20% of the sorting results.

[0062] In this possible implementation, the ratios in the comparison value sorting results are deduplicated to reduce the amount of calculation of unnecessary data and improve calculation efficiency. By determining two positions in the sorting results, the ratio in the middle of the sorting results is selected as the target ratio to remove the influence of noise in the actual data processing process.

[0063] In a possible implementation, determining a first feature point corresponding to the pixel point according to the intrinsic parameter and the RGB data, and determining a second feature point corresponding to the pixel point according to the intrinsic parameter and the depth data can be performed according to the following steps: determining a target pixel point in the RGB data; determining a first feature point corresponding to the target pixel point according to the intrinsic parameter and the RGB data, and determining a second feature point corresponding to the target pixel point according to the intrinsic parameter and the depth data.

[0064] In this possible implementation, the target pixel point is a pixel point at a specified position or a pixel point obtained by a specified method. For example, some key points with significant features and easy to detect and describe are screened from the pixel points of the RGB data, which may be places with large changes or uniqueness in the image, such as corner points, edge points, texture points, etc., to obtain the target pixel point.

[0065] In this possible implementation, the first feature point and the second feature point corresponding to the target pixel point are calculated, so as to analyze and calculate the key points to obtain a more accurate posture scale.

[0066] In a possible implementation, determining the target pixel in the RGB data may be performed according to the following steps: extracting SIFT feature points and / or ORB feature points from the RGB data; and using the SIFT feature points and / or ORB feature points as the target pixel in the RGB data.

[0067] In this possible implementation, SIFT (Scale-Invariant Feature Transform) is an algorithm for detecting and describing local features of an image. SIFT feature points are feature points extracted by the SIFT algorithm. SIFT feature points are scale-invariant and rotation-invariant, which means that they remain stable under image scaling, rotation, and partial viewing angle changes. ORB (Oriented FAST and Rotated BRIEF) is an efficient feature point detection and description algorithm. While maintaining high efficiency, the ORB algorithm has good performance and robustness and is suitable for real-time applications. ORB feature points are feature points extracted by the ORB algorithm.

[0068] In this possible implementation, SIFT feature points and / or ORB feature points of the RGB data are extracted; the SIFT feature points and / or ORB feature points are used as target pixel points in the RGB data, thereby improving the stability of the obtained pose scale.

[0069] The implementation of this method is described below in a specific implementation manner. Figure 3The implementation flow chart of the pose alignment method shown in Figure 3 In part A of the algorithm, first, we obtain the RGB data sequence (a set of RGB images arranged in time order, which can be regarded as a video or a series of continuously shot image frames) and its corresponding internal parameters. Based on the image feature points (SIFT, ORB, etc.) and the input internal parameters, we use a nonlinear optimization method to obtain the pose (rotation matrix + translation matrix) when each photo was taken. In addition, we also obtain the 2D feature points (pixel coordinates) FEATURE on each RGB photo. 2D_POSE And the 3D point FEATURE corresponding to the 2D feature point 3D_POSE (XYZ). The above results constitute the three-dimensional coordinate system of the pose.

[0070] Afterwards, see Figure 3 In part B, the depth data sequence (DEPTH map) and internal parameters are obtained to obtain the three-dimensional point cloud POINT of each depth map based on the camera coordinate system. 3D_DEPTH The RGBD camera described in the embodiment of the present application is based on structured light, with a depth perception range of 0.3 meters to 3 meters, and the acquired point cloud is a dense point cloud. The above results constitute the depth map coordinate system.

[0071] See also Figure 2 The scene diagram shown is Figure 2 The shape composed of 3D points is shown in the figure. The red structure around the shape represents the camera. The three-dimensional coordinate system of the pose and the three-dimensional coordinate system of each depth map are shown in the figure. Figure 2 As shown: The coordinate system alignment can be completed by converting the depth in each depth map coordinate system into the pose coordinate system. The difference between the two coordinate systems is a scalar scale SCALE DEPTH_TO_POSE . See Figure 3 In the C section, the following describes the coordinate system alignment process, which involves how to obtain the scalar scale SCALE DEPTH_TO_POSE .

[0072] a. According to the 2D feature point FEATURE in each RGB image in the three-dimensional coordinate system of the pose 2D_POSE And its corresponding 3D feature point FEATURE 3D_POSE , get the 3D point sequence VEC_FEATURE in the three-dimensional coordinate system of the pose 3D_POSE ; b. Obtain the pixel coordinates in the pixel coordinate system based on the 2D feature points in each RGB image in the three-dimensional coordinate system of the pose, and obtain the depth based on the pixel coordinates, depth map, and internal parameters. Figure 3 3D point sequence VEC_POINT in the 3D coordinate system 3D_DEPTH; c. The 3D point sequence needs to be real and valid. Based on this constraint, the 3D point sequences in steps a and b are filtered; d. For each RGB image, the pose and depth corresponding to the 2D feature points are selected Figure 3 3D points in the 2D coordinate system, calculate the spatial distance VEC_DISTANCE between the 3D point corresponding to each 2D point and the 3D points corresponding to the other 2D points 3D_POSE 、VEC_DISTANCE 3D_DEPTH e. The spatial distance needs to be greater than 0. Based on this constraint, filter the intermediate distance pairs in step d; f. Calculate the ratio of the distance pairs in each RGB image and obtain the ratio sequence, VEC_SCALEDEPTH _TO_POSE ;in, The value range of i is [0, N1), where N1 is the size of the spatial distance sequence. g. In order to remove the influence of noise in the actual data processing process, the ratio sequence is sorted and the smallest and largest 20% are removed. At this time, the size of the spatial sequence is reduced from N1 to M2; h. Calculate the mean and standard deviation of the ratio sequence. The mean is the required scale SCALE DEPTH_TO_POSE , standard deviation STD VEC_SCALE If it is less than 0.01, it indicates that the scale S is available, where:

[0073] SCALE DEPTH_TO_POSE =∑VEC_SCALE DEPTH_TO_POSE / N2.

[0074] The above steps make full use of the corner points of the image, combined with the effective area of ​​the depth map, to obtain a sequence of available three-dimensional points. The scale can be obtained according to the ratio between the value of the true scale of the sequence and the coordinate system obtained by the pose; combined with the effective points of the three-dimensional model coordinate system obtained based on the pose of the image and the effective value of the three-dimensional data converted based on the RGBD internal reference and the depth map, the effectiveness of obtaining the scale data can be further improved, thereby improving the accuracy of obtaining the scale; in order to remove the influence of noise in the actual data processing process, first sort, remove the maximum and minimum 20%, take the average of the remaining 60% of the data, and calculate the standard deviation. Generally, a standard deviation of less than 0.01 indicates that the result is available, and the standard deviation of the real data test is generally within 0.001. The embodiment of the present application combines the three-dimensional information obtained by the RGB image corner points and the depth map to quickly, accurately and stably obtain the pose scale. This process does not require human participation, ensures full process automation, and improves data processing efficiency; after actual verification, this method can resist the influence of a certain degree of noise and has high stability.

[0075] The present application also provides a three-dimensional reconstruction method. Figure 4 The flowchart of a three-dimensional reconstruction method according to an embodiment of the present application is shown. The method may include:

[0076] Step 401, obtaining aligned posture data; wherein the aligned posture data is obtained according to the aforementioned posture alignment method.

[0077] Step 402: Generate a three-dimensional reconstruction result based on the aligned posture data.

[0078] In the embodiments of the present application, 3D reconstruction refers to reconstructing the 3D shape and structure of an object or scene from a 2D image or other data. Performing 3D reconstruction based on aligned pose data can improve the reconstruction accuracy of the reconstruction result, simplify the calculation process, and improve robustness and reconstruction efficiency.

[0079] The present application provides a three-dimensional reconstruction method, which includes: obtaining aligned posture data; wherein the aligned posture data is obtained according to the above-mentioned posture alignment method; and generating a three-dimensional reconstruction result according to the aligned posture data. The posture alignment method can quickly, accurately and stably obtain the posture scale, optimize the alignment process of RGB data, and thus perform three-dimensional reconstruction based on the aligned posture data.

[0080] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a posture alignment device. Figure 5 The figure is a structural block diagram of a posture alignment device according to an embodiment of the present application, and the device may include:

[0081] The acquisition module 501 acquires the RGBD data captured by the camera and the internal parameters of the camera; wherein the RGBD data includes RGB data and depth data, and the pixels in the RGB data and the depth data correspond to each other one by one; the data module 502 determines the first feature point corresponding to the pixel point according to the internal parameters and the RGB data, and determines the second feature point corresponding to the pixel point according to the internal parameters and the depth data; wherein the first feature point is located in the three-dimensional coordinate system of the posture, and the second feature point is located in the depth Figure 3 The alignment module 503 calculates the three-dimensional coordinate system of the posture and the depth based on the pixel point, the first feature point, and the second feature point. Figure 3 The pose scale between the three-dimensional coordinate system to convert the pose three-dimensional coordinate system to the depth Figure 3 dimensional coordinate system alignment.

[0082] In a possible implementation, the three-dimensional coordinate system of the pose and the depth are calculated based on the pixel point, the first feature point, and the second feature point. Figure 3The method comprises: determining a plurality of pixels to be processed in the RGB data, determining a current pixel and a designated pixel in the plurality of pixels to be processed; wherein the designated pixel is any pixel other than the current pixel in the plurality of pixels to be processed; calculating the spatial distance between the current pixel and the designated pixel based on the first feature point and the second feature point, and calculating the spatial distance between the three-dimensional coordinate system of the posture and the depth according to the spatial distance. Figure 3 The pose scale between two dimensional coordinate systems.

[0083] In a possible implementation, calculating the spatial distance between the current pixel and the designated pixel based on the first feature point and the second feature point can be performed according to the following steps: calculating the first spatial distance between the first feature point corresponding to the current pixel and the first feature point corresponding to the designated pixel, and calculating the second spatial distance between the second feature point corresponding to the current pixel and the second feature point corresponding to the designated pixel.

[0084] In a possible implementation, the position three-dimensional coordinate system and the depth are calculated according to the spatial distance. Figure 3 The posture scale between the three-dimensional coordinate system can be performed according to the following steps: calculating the ratio of the first spatial distance to the second spatial distance, and determining the posture three-dimensional coordinate system and the depth according to the ratio. Figure 3 The pose scale between two dimensional coordinate systems.

[0085] In a possible implementation, the three-dimensional coordinate system of the pose and the depth are calculated based on the pixel point, the first feature point, and the second feature point. Figure 3 Before the posture scale between the two-dimensional coordinate systems, it also includes: sorting the first feature points corresponding to the RGB data to obtain a first point sequence, and sorting the second feature points corresponding to the RGB data according to the pixel point order corresponding to the first point sequence to obtain a second point sequence; determining the pixel point to be processed among the multiple pixel points to be processed, including: determining the pixel point to be processed among the multiple pixel points to be processed according to the pixel point order corresponding to the first point sequence.

[0086] In a possible implementation manner, the three-dimensional coordinate system of the posture and the depth are determined according to the ratio. Figure 3 The method comprises: sorting the ratios to obtain a ratio sorting result; determining a plurality of target ratios in the ratio sorting result according to a preset ratio screening condition, and calculating the mean and standard deviation of the plurality of target ratios; when the standard deviation is less than a preset standard deviation threshold, using the mean as the ratio between the three-dimensional coordinate system of the posture and the depth. Figure 3The pose scale between two dimensional coordinate systems.

[0087] In one possible implementation, multiple target ratios are determined in the ratio sorting results based on preset ratio screening conditions, including: deduplicating the ratios in the ratio sorting results; taking the ratio between the first position and the second position in the ratio sorting results after deduplication as the target ratio; wherein the first position is before the second position.

[0088] In a possible implementation, determining a first feature point corresponding to the pixel point according to the intrinsic parameter and the RGB data, and determining a second feature point corresponding to the pixel point according to the intrinsic parameter and the depth data, includes: determining a target pixel point in the RGB data; determining the first feature point corresponding to the target pixel point according to the intrinsic parameter and the RGB data, and determining the second feature point corresponding to the target pixel point according to the intrinsic parameter and the depth data.

[0089] In a possible implementation, determining the target pixel in the RGB data includes: extracting SIFT feature points and / or ORB feature points from the RGB data; and using the SIFT feature points and / or ORB feature points as the target pixel in the RGB data.

[0090] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a three-dimensional reconstruction device. The device may include: a posture module, used to obtain aligned posture data; wherein the aligned posture data is obtained according to the above-mentioned posture alignment method; and a reconstruction module, used to generate a three-dimensional reconstruction result according to the aligned posture data.

[0091] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0092] Figure 6 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 6 As shown, the electronic device includes: a memory 601 and a processor 602. The memory 601 stores a computer program that can be run on the processor 602. When the processor 602 executes the computer program, the method in the above embodiment is implemented. The number of the memory 601 and the processor 602 can be one or more.

[0093] The electronic device also includes:

[0094] The communication interface 603 is used to communicate with external devices and perform data exchange transmission.

[0095] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the memory 601, the processor 602 and the communication interface 603 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0097] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0098] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0099] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0100] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0101] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0102] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0103] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0106] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0107] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0108] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0109] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0110] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A pose alignment method, wherein: include: Acquire RGBD data captured by a camera and the internal parameters of the camera; wherein the RGBD data includes RGB data and depth data, and the pixels in the RGB data and the depth data correspond one to one; Determine the first feature point corresponding to the pixel point according to the intrinsic parameter and the RGB data, and determine the pixel point according to the intrinsic parameter and the depth data. Determine a second feature point corresponding to the pixel point; wherein the first feature point is located in a three-dimensional coordinate system of a posture, and the second feature point is located in a three-dimensional coordinate system of a depth map; A pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system is calculated based on the pixel points, the first feature points, and the second feature points to align the pose three-dimensional coordinate system with the depth map three-dimensional coordinate system.

2. The method according to claim 1, wherein: Calculating a pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system based on the pixel point, the first feature point, and the second feature point includes: Determine a plurality of pixels to be processed in the RGB data, and determine a current pixel and a designated pixel among the plurality of pixels to be processed; wherein the designated pixel is any pixel among the plurality of pixels to be processed except the current pixel; The spatial distance between the current pixel and the designated pixel is calculated based on the first feature point and the second feature point, and the pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system is calculated according to the spatial distance.

3. The method according to claim 2, wherein: Calculating the spatial distance between the current pixel and the designated pixel based on the first feature point and the second feature point includes: A first spatial distance between a first feature point corresponding to the current pixel and a first feature point corresponding to the designated pixel is calculated, and a second spatial distance between a second feature point corresponding to the current pixel and a second feature point corresponding to the designated pixel is calculated.

4. The method according to claim 3, wherein: Calculating the pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system according to the spatial distance includes: A ratio of the first spatial distance to the second spatial distance is calculated, and a pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system is determined according to the ratio.

5. The method according to claim 4, wherein: Determining a pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system according to the ratio includes: Sorting the ratios to obtain a ratio sorting result; According to a preset ratio screening condition, multiple target ratios are determined in the ratio sorting results, and the mean and standard deviation of the multiple target ratios are calculated; When the standard deviation is less than a preset standard deviation threshold, the mean is used as a pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system.

6. The method according to claim 5, wherein: According to the preset ratio screening conditions, multiple target ratios are determined in the ratio sorting results, including: Deduplication processing is performed on the ratios in the ratio sorting result; The ratio between the first position and the second position in the ratio sorting result after deduplication processing is used as the target ratio; wherein the first position is located before the second position.

7. The method according to claim 2, wherein: Before calculating the pose scale between the pose three-dimensional coordinate system and the depth map three-dimensional coordinate system based on the pixel point, the first feature point, and the second feature point, the method further includes: Sorting the first feature points corresponding to the RGB data to obtain a first point sequence, and sorting the second feature points corresponding to the RGB data according to the pixel point order corresponding to the first point sequence to obtain a second point sequence; Determining a current pixel point among the plurality of pixel points to be processed includes: determining the current pixel point among the plurality of pixel points to be processed according to a pixel point sequence corresponding to the first point sequence.

8. The method according to claim 1, wherein: Determining a first feature point corresponding to the pixel point according to the intrinsic parameter and the RGB data, and determining a second feature point corresponding to the pixel point according to the intrinsic parameter and the depth data, including: Determine a target pixel in the RGB data; A first feature point corresponding to the target pixel is determined according to the intrinsic parameter and the RGB data, and a second feature point corresponding to the target pixel is determined according to the intrinsic parameter and the depth data.

9. The method according to claim 8, wherein: Determining a target pixel point in the RGB data includes: Extracting SIFT feature points and / or ORB feature points of the RGB data; The SIFT feature points and / or ORB feature points are used as target pixel points in the RGB data.

10. A three-dimensional reconstruction method, comprising: Acquire aligned posture data; wherein the aligned posture data is obtained according to the method described in any one of claims 1 to 9; Generate a three-dimensional reconstruction result according to the aligned posture data.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

12. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images