A depth map attitude correction method, system, computer device, and storage medium.

Abstract

This invention discloses a depth map pose correction method, system, computer device, and storage medium. The method includes converting pixels within a region of interest (ROI) on the depth map into a point cloud; performing plane fitting on the point cloud based on orthogonal polynomials; generating a pose correction matrix based on the fitted plane; correcting the pose of the point cloud using the pose correction matrix; determining the minimum bounding box of the point cloud; generating a rigid body transformation matrix based on the geometric center point of the bounding box and the corrected reference plane to move the lower left corner of the bounding box to the Z-axis and align the reference plane with the XY plane; recalculating the scaling ratio of the point cloud based on the size of the output image and the ratio of the bounding box; scaling the point cloud; and calculating the corresponding pixel value by weighted summation of points within each pixel using the PSF point spread function model to generate a pose-corrected depth map. This invention can avoid distortion caused by projection deformation of the data while performing pose correction on the depth map.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a depth map pose correction method, system, computer device, and storage medium. Background Technology

[0002] A depth map, also known as a depth image, is an image that uses the distance from the image acquisition device to various points on an object as pixel values.

[0003] Currently, depth map pose correction is mostly used as preprocessing for geometric measurements. Due to differences in the installation location of the 3D data acquisition unit and the actual object, or the inability to guarantee perfect parallelism between the object's surface and the camera plane, the acquired 3D data is often tilted. While general plane correction algorithms can correct tilted planes, they suffer from projection distortion, leading to inaccuracies in the measurement and calculation of the real object.

[0004] Therefore, improvements to existing technologies are necessary.

[0005] The above information is provided as background information only to aid in understanding this disclosure and does not constitute an assertion or admission that any of the above content can be used as prior art relative to this disclosure. Summary of the Invention

[0006] This invention provides a depth map attitude correction method, system, computer device, and storage medium to overcome the shortcomings of the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a depth map pose correction method, the method comprising:

[0009] The pixels within the ROI region on the depth map are converted into point clouds. The point clouds are then fitted with a plane based on orthogonal polynomials, and a pose correction matrix is ​​generated based on the fitted plane.

[0010] The point cloud is subjected to attitude correction according to the attitude correction matrix. The AABB minimum bounding box of the corrected point cloud is obtained. A rigid body transformation matrix is ​​generated according to the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and make the reference plane coincide with the XY plane.

[0011] The scaling ratio of the point cloud is recalculated based on the size of the image to be output and the ratio of the AABB minimum bounding box. The point cloud is scaled, and after scaling, the points within each pixel are weighted and summed according to the PSF point spread function model to calculate the pixel value of the corresponding point in the point cloud on the depth map, so as to generate the pose-corrected depth map.

[0012] Furthermore, in the depth map pose correction method, the steps of converting pixels within the ROI region of the depth map into point clouds, performing plane fitting based on orthogonal polynomials on the point clouds, and generating a pose correction matrix based on the fitted plane include:

[0013] Extract the Region of Interest (ROI) from the depth map and convert the pixels within the ROI into a point cloud;

[0014] An orthogonal polynomial system is constructed using the Schmidt orthogonalization method, and orthogonal basis matrices are constructed in the X and Y directions, respectively.

[0015] The point cloud is fitted with the orthogonal basis matrix to obtain the fitted plane, and the relationship between the normal vector of the fitted plane and the world coordinate Z-axis is calculated to generate the attitude correction matrix.

[0016] Further, in the depth map attitude correction method, the step of performing attitude correction on the point cloud according to the attitude correction matrix, obtaining the AABB minimum bounding box of the corrected point cloud, and generating a rigid body transformation matrix based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z-axis and make the reference plane coincide with the XY plane, includes:

[0017] The point cloud is subjected to attitude correction based on the attitude correction matrix to obtain the corrected point cloud.

[0018] The minimum bounding box of the corrected point cloud is quickly obtained by using TBB multithreading.

[0019] Based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, a 4x4 rigid body transformation matrix is ​​generated to move the lower left corner of the AABB minimum bounding box onto the Z-axis and move the rotationally corrected reference plane to a height that coincides with the XY plane.

[0020] Furthermore, in the depth map pose correction method, the steps of recalculating the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scaling the point cloud, and then, after scaling, calculating the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model, to generate the pose-corrected depth map, include:

[0021] The scaling ratio of the point cloud is recalculated based on the width and height of the image to be output and the length and width of the minimum bounding box of the AABB in the XY direction, and the point cloud is scaled to the size of the image to be output.

[0022] After scaling, based on the PSF point spread function model, the TBB segmentation and merging strategy is used to calculate the weights and sums of points within each block of the image pixels, and the pixel values ​​of the corresponding points in the point cloud on the depth map are extracted to generate the pose-corrected depth map.

[0023] Furthermore, in the depth map pose correction method, after the steps of recalculating the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scaling the point cloud, and then, after scaling, calculating the pixel value of the corresponding point in the point cloud on the depth map by weighted summation based on the PSF point spread function model, in order to generate a pose-corrected depth map, the method further includes:

[0024] Fill in invalid value points between pixels in the depth map after pose correction.

[0025] Furthermore, in the depth map pose correction method, the step of filling in invalid value points between pixels in the pose-corrected depth map includes:

[0026] Using the orthogonal polynomial system, the first n and m basis functions are taken respectively to construct a local spline polynomial basis matrix BL for surface fitting, and the fitted surface S is obtained.

[0027] The depth map is traversed using TBB multithreading, and the invalid value points between pixels in the pose-corrected depth map are filled with the corresponding pixels of the fitted surface S using the fitted surface S, while the valid value points remain unchanged.

[0028] Furthermore, in the depth map pose correction method, the step of filling in invalid value points between pixels in the pose-corrected depth map includes:

[0029] The depth map is traversed using TBB multithreading, and invalid value points between pixels in the pose-corrected depth map are filled with a 5x5 kernel for mean filling.

[0030] In a second aspect, the present invention provides a depth map attitude correction system, the system comprising:

[0031] The matrix generation module is used to convert pixels in the ROI region on the depth map into point clouds, perform plane fitting based on orthogonal polynomials on the point clouds, and generate a pose correction matrix based on the fitted plane.

[0032] The attitude correction module is used to perform attitude correction on the point cloud according to the attitude correction matrix, obtain the AABB minimum bounding box of the corrected point cloud, and generate a rigid body transformation matrix according to the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and make the reference plane coincide with the XY plane.

[0033] The scaling calculation module is used to recalculate the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scale the point cloud, and after scaling, calculate the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model, so as to generate the pose-corrected depth map.

[0034] Furthermore, in the depth map pose correction system, the matrix generation module is specifically used for:

[0035] Extract the Region of Interest (ROI) from the depth map and convert the pixels within the ROI into a point cloud;

[0036] An orthogonal polynomial system is constructed using the Schmidt orthogonalization method, and orthogonal basis matrices are constructed in the X and Y directions, respectively.

[0037] The point cloud is fitted with the orthogonal basis matrix to obtain the fitted plane, and the relationship between the normal vector of the fitted plane and the world coordinate Z-axis is calculated to generate the attitude correction matrix.

[0038] Furthermore, in the depth map attitude correction system, the attitude correction module is specifically used for:

[0039] The point cloud is subjected to attitude correction based on the attitude correction matrix to obtain the corrected point cloud.

[0040] The minimum bounding box of the corrected point cloud is quickly obtained by using TBB multithreading.

[0041] Based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, a 4x4 rigid body transformation matrix is ​​generated to move the lower left corner of the AABB minimum bounding box onto the Z-axis and move the rotationally corrected reference plane to a height that coincides with the XY plane.

[0042] Furthermore, in the depth map pose correction system, the scaling calculation module is specifically used for:

[0043] The scaling ratio of the point cloud is recalculated based on the width and height of the image to be output and the length and width of the minimum bounding box of the AABB in the XY direction, and the point cloud is scaled to the size of the image to be output.

[0044] After scaling, based on the PSF point spread function model, the TBB segmentation and merging strategy is used to calculate the weights and sums of points within each block of the image pixels, and the pixel values ​​of the corresponding points in the point cloud on the depth map are extracted to generate the pose-corrected depth map.

[0045] Furthermore, in the depth map pose correction system, the system also includes an invalid fill module, used for:

[0046] After the steps of recalculating the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scaling the point cloud, and then calculating the pixel value of the corresponding point in the depth map based on the weighted summation of the points within each pixel according to the PSF point spread function model to generate the pose-corrected depth map, invalid value points between pixels in the pose-corrected depth map are filled.

[0047] Furthermore, in the depth map pose correction system, the invalid fill module is specifically used for:

[0048] Using the orthogonal polynomial system, the first n and m basis functions are taken respectively to construct a local spline polynomial basis matrix BL for surface fitting, and the fitted surface S is obtained.

[0049] The depth map is traversed using TBB multithreading, and the fitted surface S is used to fill invalid value points between pixels in the pose-corrected depth map with S(i,j), while the valid value points remain unchanged.

[0050] Furthermore, in the depth map pose correction system, the invalid fill module is specifically used for:

[0051] The depth map is traversed using TBB multithreading, and invalid value points between pixels in the pose-corrected depth map are filled with a 5x5 kernel for mean filling.

[0052] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the depth map pose correction method as described in the first aspect above.

[0053] Fourthly, the present invention provides a storage medium containing computer-executable instructions, which are executed by a computer processor to implement the depth map pose correction method as described in the first aspect above.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention provides a depth map attitude correction method, system, computer device and storage medium. By performing attitude correction on the depth map based on plane fitting of orthogonal polynomials, the distortion caused by the projection deformation of the data can be avoided during the correction. It is of great significance to adopt and promote. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic flowchart of a depth map attitude correction method provided in Embodiment 1 of the present invention;

[0058] Figure 2 This is a schematic diagram of the 3D camera mentioned in Embodiment 1 of the present invention;

[0059] Figure 3 This is a schematic diagram of workpiece depth data acquisition as mentioned in Embodiment 1 of the present invention;

[0060] Figure 4 This is a schematic diagram of the imported depth map mentioned in Embodiment 1 of the present invention;

[0061] Figure 5 This is a schematic diagram of the process mentioned in Embodiment 1 of the present invention, in which the lower left corner point of the corrected point cloud data bounding box is moved to the Z-axis and the reference plane after rotation correction is moved to a height that coincides with the XY plane.

[0062] Figure 6 This is a schematic diagram of depth map pose correction mentioned in Embodiment 1 of the present invention;

[0063] Figure 7 This is a schematic diagram of depth map pose correction mentioned in Embodiment 1 of the present invention;

[0064] Figure 8 This is a schematic diagram of the invalid value point filling effect mentioned in Embodiment 1 of the present invention;

[0065] Figure 9 This is a schematic diagram of the filling effect of the neighborhood mean filling of invalid points mentioned in Embodiment 1 of the present invention;

[0066] Figure 10 This is a schematic diagram of the moving window method combining TBB and SIMD mentioned in Embodiment 1 of the present invention.

[0067] Figure 11This is a schematic diagram of the functional modules of a depth map attitude correction system provided in Embodiment 2 of the present invention;

[0068] Figure 12 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0069] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0070] In the description of this invention, it should be understood that when a component is considered to be "connected" to another component, it can be directly connected to the other component or there may be an intermediate component present simultaneously. When a component is considered to be "set" on another component, it can be directly set on the other component or there may be an intermediate component present simultaneously.

[0071] Furthermore, terms such as “long,” “short,” “inner,” and “outer” indicate orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings. They are used only for the purpose of describing the present invention and are not intended to indicate or imply that the device or component referred to must have this specific orientation or operate in a specific orientational configuration. Therefore, they should not be construed as limitations of the present invention.

[0072] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0073] Example 1

[0074] In view of the shortcomings of existing depth map attitude correction technologies, the applicant, based on years of practical experience and professional knowledge in this field, and combined with the application of theoretical principles, has actively researched and innovated to create a technology that can solve the shortcomings of existing technologies and make depth map attitude correction technology more practical. After continuous research, design, and repeated prototype production and improvement, this invention with real practical value has finally been created.

[0075] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a depth map pose correction method according to Embodiment 1 of the present invention. This method is applicable to scenarios requiring pose correction of depth maps. The method is executed by a depth map pose correction system, which can be implemented in software and / or hardware. The method specifically includes the following steps:

[0076] S101. Convert the pixels in the ROI region on the depth map into a point cloud, perform plane fitting based on orthogonal polynomials on the point cloud, and generate a pose correction matrix based on the fitted plane.

[0077] It should be noted that this embodiment uses, as shown in the example... Figure 2 The 3D camera captures images as shown. Figure 3 The workpiece depth data shown enables the detection and measurement of the workpiece.

[0078] In this embodiment, step S101 can be further refined to include the following steps:

[0079] Extract the Region of Interest (ROI) from the depth map and convert the pixels within the ROI into a point cloud;

[0080] An orthogonal polynomial system is constructed using the Schmidt orthogonalization method, and orthogonal basis matrices are constructed in the X and Y directions, respectively.

[0081] The point cloud is fitted with the orthogonal basis matrix to obtain the fitted plane, and the relationship between the normal vector of the fitted plane and the world coordinate Z-axis is calculated to generate the attitude correction matrix.

[0082] It should be noted that importing depth maps, such as... Figure 4 As shown, an orthogonal polynomial system with respect to the weight function w(x)=1 on the interval [-1,1] is constructed using the Schmidt orthogonalization method.

[0083] ;

[0084] Edit the region of interest (ROI) on the depth map, extract the set of ground truth points from the ROI, and take the value from the formula above. Construct two orthogonal basis matrices of order 2 in the X and Y directions, respectively, Bx={ },By= Perform plane fitting to obtain the fitted plane P.

[0085] After obtaining the fitted plane P, calculate the angles α and β between the normal of the plane and the Z-axis after rotation around the X and Y axes, and generate the correction matrix M.

[0086] ;

[0087] Where α and β are the rotation angles about the X-axis and Y-axis, respectively.

[0088] S102. Perform attitude correction on the point cloud according to the attitude correction matrix, obtain the AABB minimum bounding box of the corrected point cloud, and generate a rigid body transformation matrix according to the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and make the reference plane coincide with the XY plane.

[0089] In this embodiment, step S102 can be further refined to include the following steps:

[0090] The point cloud is subjected to attitude correction based on the attitude correction matrix to obtain the corrected point cloud.

[0091] The minimum bounding box of the corrected point cloud is quickly obtained by using TBB multithreading.

[0092] Based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, a 4x4 rigid body transformation matrix is ​​generated to move the lower left corner of the AABB minimum bounding box onto the Z-axis and move the rotationally corrected reference plane to a height that coincides with the XY plane.

[0093] It should be noted that the AABB minimum bounding box of the corrected point cloud is obtained. To quickly determine the maximum and minimum point coordinates in the X / Y / Z directions, the six parameters of the AABB minimum bounding box can be expressed as follows: and The geometric center of the AABB bounding box is Multi-threaded computation can be performed using TBB's parallel_reduce method to divide and calculate the data of each sub-block. Finally, the results of each sub-block are merged to obtain the minimum bounding box of the point cloud. Then, a 4x4 rigid body transformation matrix is ​​generated based on the X and Y coordinates of the geometric center point of the minimum bounding box and the Z coordinate of the corrected reference plane. This matrix moves the lower left corner of the corrected point cloud bounding box to the Z-axis and moves the rotated and corrected reference plane to a height coinciding with the XY plane. Figure 5 As shown.

[0094] S103. Based on the size of the image to be output and the ratio of the AABB minimum bounding box, recalculate the scaling ratio of the point cloud, scale the point cloud, and after scaling, calculate the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model, so as to generate the pose-corrected depth map.

[0095] In this embodiment, step S103 can be further refined to include the following steps:

[0096] The scaling ratio of the point cloud is recalculated based on the width and height of the image to be output and the length and width of the minimum bounding box of the AABB in the XY direction, and the point cloud is scaled to the size of the image to be output.

[0097] After scaling, based on the PSF point spread function model, the TBB segmentation and merging strategy is used to calculate the weights and sums of points within each block of the image pixels, and the pixel values ​​of the corresponding points in the point cloud on the depth map are extracted to generate the pose-corrected depth map.

[0098] It should be noted that this step is for converting point clouds to depth maps. Based on the width and height of the image to be output and the length and width of the minimum bounding box in the XY directions, the scaling ratio of the point cloud is recalculated using the following formula: and The point cloud is scaled to the size of the image, and the corresponding pixel value is calculated by weighted summation of the points within each pixel using the PSF point spread function model. The PSF point spread model formula is as follows: Point cloud to depth map conversion based on PSF requires converting the point cloud into a depth map after pose correction. This involves calculating the value mapped from the point cloud for each pixel. The principle of Gaussian point spread function is used here to weight the z-value of each point within a pixel to obtain the corresponding pixel value.

[0099] In this embodiment, it should be noted that the image will become larger as it changes from a slope to a front view, and there may be some holes (i.e. invalid value points) between pixels. These invalid value points need to be quickly filled after the depth map is corrected.

[0100] That is, after step S103, the method can be further refined to include the following steps:

[0101] Fill in invalid value points between pixels in the depth map after pose correction.

[0102] Next, this embodiment will introduce two implementation methods for filling invalid value points.

[0103] In the first embodiment, the step of filling in invalid value points between pixels in the pose-corrected depth map can be further refined to include the following steps:

[0104] Using the orthogonal polynomial system described in the above steps, the first n and m basis functions are taken respectively to construct a local spline polynomial basis matrix BL for surface fitting, and the fitted surface S is obtained. The orthogonal basis of the locally constructed orthogonal polynomial can solve the Runge phenomenon problem in high-order global data fitting.

[0105] The depth map is traversed using TBB multithreading, and the invalid value points between pixels in the pose-corrected depth map are filled with the corresponding pixels of the fitted surface S using the fitted surface S, while the valid value points remain unchanged.

[0106] In the second embodiment, the step of filling in invalid value points between pixels in the pose-corrected depth map can be further refined to include the following steps:

[0107] The depth map is traversed using TBB multithreading, and invalid value points between pixels in the pose-corrected depth map are filled with a 5x5 kernel for mean filling.

[0108] It should be noted that this embodiment can also quickly fill invalid points by using a 5x5 kernel for mean filling on the pose-corrected depth map. TBB multithreading is used to traverse the depth map, with invalid points set to 0 and valid points set to 1. First, TBB multithreading is used to calculate the positions of all invalid points in the depth map. Then, a moving window method combining TBB and SIMD is used to calculate the number of invalid points in the neighborhood of each point. Since the moving window only needs to add and subtract a row (column) of data in the X and Y directions when calculating the neighborhood, it avoids calculating all pixels within the window, reducing redundant calculations. Figure 10 As shown, moving the window avoids the drastic speed drop caused by an excessively large window. Finally, invalid points are filled with the neighborhood mean, and the filling effect is as follows. Figure 9 As shown, discrete invalid value points within the target are filled with the mean of their neighborhood. The depth map pose correction and invalid value point filling effect is as follows: Figure 6-8 As shown.

[0109] Although this paper frequently uses terms such as depth map, pose, correction, bounding box, matrix, and scaling ratio, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

[0110] This invention provides a depth map attitude correction method that performs attitude correction on the depth map by plane fitting based on orthogonal polynomials. This method can avoid distortion caused by projection deformation of the data while correcting the attitude. In addition, by using the depth map as the processing object, it adopts TBB multi-threaded parallelism combined with SIMD multi-data parallelism to improve the computation speed of the algorithm. It can handle the correction needs in industrial 3D projects in real time and has great significance for adoption and promotion.

[0111] Example 2

[0112] Please refer to Figure 11 This is a functional module diagram of a depth map attitude correction system provided in Embodiment 5 of the present invention. This system is applicable to executing the depth map attitude correction method provided in this embodiment of the present invention. Specifically, the system includes the following modules:

[0113] The matrix generation module 201 is used to convert the pixels in the ROI region on the depth map into a point cloud, perform plane fitting based on orthogonal polynomials on the point cloud, and generate a pose correction matrix based on the fitted plane.

[0114] The attitude correction module 202 is used to perform attitude correction on the point cloud according to the attitude correction matrix, obtain the AABB minimum bounding box of the corrected point cloud, and generate a rigid body transformation matrix according to the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and make the reference plane coincide with the XY plane.

[0115] The scaling calculation module 203 is used to recalculate the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scale the point cloud, and after scaling, calculate the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model, so as to generate the pose-corrected depth map.

[0116] Preferably, the matrix generation module 201 is specifically used for:

[0117] Extract the Region of Interest (ROI) from the depth map and convert the pixels within the ROI into a point cloud;

[0118] An orthogonal polynomial system is constructed using the Schmidt orthogonalization method, and orthogonal basis matrices are constructed in the X and Y directions, respectively.

[0119] The point cloud is fitted with the orthogonal basis matrix to obtain the fitted plane, and the relationship between the normal vector of the fitted plane and the world coordinate Z-axis is calculated to generate the attitude correction matrix.

[0120] Preferably, the attitude correction module 202 is specifically used for:

[0121] The point cloud is subjected to attitude correction based on the attitude correction matrix to obtain the corrected point cloud.

[0122] The minimum bounding box of the corrected point cloud is quickly obtained by using TBB multithreading.

[0123] Based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, a 4x4 rigid body transformation matrix is ​​generated to move the lower left corner of the AABB minimum bounding box onto the Z-axis and move the rotationally corrected reference plane to a height that coincides with the XY plane.

[0124] Preferably, the scaling calculation module 203 is specifically used for:

[0125] The scaling ratio of the point cloud is recalculated based on the width and height of the image to be output and the length and width of the minimum bounding box of the AABB in the XY direction, and the point cloud is scaled to the size of the image to be output.

[0126] After scaling, based on the PSF point spread function model, the TBB segmentation and merging strategy is used to calculate the weights and sums of points within each block of the image pixels, and the pixel values ​​of the corresponding points in the point cloud on the depth map are extracted to generate the pose-corrected depth map.

[0127] Preferably, the method further includes an invalid padding module, used for:

[0128] After the steps of recalculating the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scaling the point cloud, and then calculating the pixel value of the corresponding point in the depth map based on the weighted summation of the points within each pixel according to the PSF point spread function model to generate the pose-corrected depth map, invalid value points between pixels in the pose-corrected depth map are filled.

[0129] Preferably, the invalid fill module is specifically used for:

[0130] Using the orthogonal polynomial system, the first n and m basis functions are taken respectively to construct a local spline polynomial basis matrix BL for surface fitting, and the fitted surface S is obtained.

[0131] The depth map is traversed using TBB multithreading, and the invalid value points between pixels in the pose-corrected depth map are filled with the corresponding pixels of the fitted surface S using the fitted surface S, while the valid value points remain unchanged.

[0132] Preferably, the invalid fill module is specifically used for:

[0133] The depth map is traversed using TBB multithreading, and invalid value points between pixels in the pose-corrected depth map are filled with a 5x5 kernel for mean filling.

[0134] The present invention provides a depth map attitude correction system that performs attitude correction on the depth map by plane fitting based on orthogonal polynomials. It can avoid distortion caused by projection deformation of data while correcting the image, and has great significance for adoption and promotion.

[0135] The above system can execute the methods provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the methods.

[0136] Example 3

[0137] Figure 12 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 12A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 12 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0138] like Figure 12 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0139] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0140] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0141] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 12 Not shown; usually referred to as a "hard drive"). Although Figure 12 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0142] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0143] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and / or with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although... Figure 12 As not shown, it can be used in conjunction with computer device 12 with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the depth map pose correction method provided in the embodiments of the present invention.

[0145] Example 4

[0146] Embodiment 4 of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the depth map pose correction method provided in all embodiments of the present application.

[0147] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0148] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0149] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0150] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this application is intended to encompass various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this application and are within the spirit and scope of the exemplary embodiments of this application.

[0152] Furthermore, certain terms used in this application have been used to describe embodiments of this application. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this application. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be appropriately combined in one or more embodiments of this application.

[0153] It should be understood that in the foregoing description of the embodiments of this application, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may extract some features as separate embodiments when reading this application. That is, the embodiments in this application can also be understood as an integration of multiple sub-embodiments. It is also valid when the content of each sub-embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0154] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. The entire contents used for all purposes, except for any history of prosecution documents associated with it, that may be inconsistent with or conflict with this document, or that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0155] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments of this application. Other modified embodiments are also within the scope of this application. Therefore, the embodiments disclosed herein are merely examples and not limitations. Those skilled in the art can adopt alternative configurations to implement the applications in this application based on the embodiments in this application. Therefore, the embodiments of this application are not limited to the embodiments precisely described in the application.

Claims

1. A depth map attitude correction method, characterized in that, The method includes: The pixels within the ROI region on the depth map are converted into point clouds. The point clouds are then fitted with a plane based on orthogonal polynomials, and a pose correction matrix is ​​generated based on the fitted plane. The point cloud is subjected to attitude correction according to the attitude correction matrix. The AABB minimum bounding box of the corrected point cloud is obtained. A rigid body transformation matrix is ​​generated according to the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and make the reference plane coincide with the XY plane. The scaling ratio of the point cloud is recalculated based on the size of the image to be output and the ratio of the AABB minimum bounding box. The point cloud is scaled, and after scaling, the points within each pixel are weighted and summed according to the PSF point spread function model to calculate the pixel value of the corresponding point in the point cloud on the depth map, so as to generate the pose-corrected depth map.

2. The depth map attitude correction method according to claim 1, characterized in that, The steps of converting pixels within the ROI region on the depth map into a point cloud, performing plane fitting on the point cloud based on orthogonal polynomials, and generating a pose correction matrix based on the fitted plane include: Extract the Region of Interest (ROI) from the depth map and convert the pixels within the ROI into a point cloud; An orthogonal polynomial system is constructed using the Schmidt orthogonalization method, and orthogonal basis matrices are constructed in the X and Y directions, respectively. The point cloud is fitted with the orthogonal basis matrix to obtain the fitted plane, and the relationship between the normal vector of the fitted plane and the world coordinate Z-axis is calculated to generate the attitude correction matrix.

3. The depth map attitude correction method according to claim 2, characterized in that, The steps of performing attitude correction on the point cloud according to the attitude correction matrix, obtaining the AABB minimum bounding box of the corrected point cloud, and generating a rigid body transformation matrix based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, to move the lower left corner point of the AABB minimum bounding box to the Z-axis and align the reference plane with the XY plane, include: The point cloud is subjected to attitude correction based on the attitude correction matrix to obtain the corrected point cloud. The minimum bounding box of the corrected point cloud is quickly obtained by TBB multithreading. A 4x4 rigid body transformation matrix is ​​generated based on the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and move the rotationally corrected reference plane to a height that coincides with the XY plane.

4. The depth map attitude correction method according to claim 3, characterized in that, The steps of recalculating the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scaling the point cloud, and then calculating the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model after scaling to generate a pose-corrected depth map include: The scaling ratio of the point cloud is recalculated based on the width and height of the image to be output and the length and width of the minimum bounding box of the AABB in the XY direction, and the point cloud is scaled to the size of the image to be output. After scaling, based on the PSF point spread function model, the TBB segmentation and merging strategy is used to calculate the weights and sums of points within each block of the image pixels, and the pixel values ​​of the corresponding points in the point cloud on the depth map are extracted to generate the pose-corrected depth map.

5. The depth map attitude correction method according to claim 4, characterized in that, After the steps of recalculating the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scaling the point cloud, and then calculating the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model after scaling, in order to generate a pose-corrected depth map, the method further includes: Fill in invalid value points between pixels in the depth map after pose correction.

6. The depth map attitude correction method according to claim 5, characterized in that, The step of filling in invalid value points between pixels in the pose-corrected depth map includes: Using the orthogonal polynomial system, the first n and m basis functions are taken respectively to construct the local spline polynomial basis matrix BL for surface fitting, and the fitted surface S is obtained. The depth map is traversed using TBB multithreading, and the invalid value points between pixels in the pose-corrected depth map are filled with the corresponding pixels of the fitted surface S using the fitted surface S, while the valid value points remain unchanged.

7. The depth map attitude correction method according to claim 5, characterized in that, The step of filling in invalid value points between pixels in the pose-corrected depth map includes: The depth map is traversed using TBB multithreading, and invalid value points between pixels in the pose-corrected depth map are filled with a 5x5 kernel for mean filling.

8. A depth map attitude correction system, characterized in that, The system includes: The matrix generation module is used to convert pixels in the ROI region on the depth map into point clouds, perform plane fitting based on orthogonal polynomials on the point clouds, and generate a pose correction matrix based on the fitted plane. The attitude correction module is used to perform attitude correction on the point cloud according to the attitude correction matrix, obtain the AABB minimum bounding box of the corrected point cloud, and generate a rigid body transformation matrix according to the X and Y coordinates of the geometric center point of the AABB minimum bounding box and the Z coordinate of the corrected reference plane, so as to move the lower left corner point of the AABB minimum bounding box to the Z axis and make the reference plane coincide with the XY plane. The scaling calculation module is used to recalculate the scaling ratio of the point cloud based on the size of the image to be output and the ratio of the AABB minimum bounding box, scale the point cloud, and after scaling, calculate the pixel value of the corresponding point in the point cloud on the depth map by weighted summation according to the PSF point spread function model, so as to generate the pose-corrected depth map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the depth map attitude correction method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, which are executed by a computer processor to implement the depth map attitude correction method as described in any one of claims 1-7.

Images