Depth map processing method and device, electronic device and readable storage medium

Abstract

The present application discloses a depth map processing method and device, an electronic device and a readable storage medium, and belongs to the field of image processing technology. The depth map processing method includes: obtaining multiple original depth maps of the target scene at multiple shooting angles, and the multiple shooting angles correspond to the multiple original depth maps one by one; according to the shooting parameters of the multiple original depth maps, a pixel matching relationship between every two original depth maps in the multiple original depth maps is established; according to the preset correction model and the pixel matching relationship, the pixel depth values ​​of the multiple original depth maps are adjusted. The embodiment of the present application corrects the obtained depth map through an automated correction model, thereby correcting the measurement error of the depth map caused by the ambient temperature, ensuring the accuracy of the distance measurement of each viewing angle and the matching precision, thereby improving the accuracy of the point cloud data.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a depth map processing method and device, an electronic device, and a readable storage medium. Background Art

[0002] In the related art, when a depth camera is used to obtain a depth map to scan and measure the distance of a scene structure, the distance measurement information in the captured depth map is affected by the ambient temperature, and the distance measurement result is deflected around a certain spatial axis, resulting in inconsistent distance measurement information for the same position under different viewing angles, thereby reducing the accuracy of the depth map display and thus reducing the accuracy of the point cloud data. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a depth map processing method and device, an electronic device and a readable storage medium, which can solve the problem in the prior art that the ranging information in the depth map is affected by the ambient temperature, thereby reducing the accuracy of the depth map display and the precision of the point cloud data.

[0004] In a first aspect, an embodiment of the present application provides a depth map processing method, the method comprising: obtaining multiple original depth maps of a target scene at multiple shooting angles, the multiple shooting angles and the multiple original depth maps Figure 1 One-to-one correspondence; establishing a pixel matching relationship between every two original depth maps in the multiple original depth maps according to the shooting parameters of the multiple original depth maps; adjusting the pixel depth values ​​of the multiple original depth maps according to the preset correction model and the pixel matching relationship.

[0005] In a second aspect, an embodiment of the present application provides a depth map processing device, the device comprising: an acquisition unit for acquiring a plurality of original depth maps of a target scene at a plurality of shooting angles, the plurality of shooting angles and the plurality of original depth maps; Figure 1 One-to-one correspondence; a processing unit, used to establish a pixel matching relationship between every two original depth maps in the multiple original depth maps according to the shooting parameters of the multiple original depth maps; the processing unit is also used to adjust the pixel depth values ​​of the multiple original depth maps according to the preset correction model and the pixel matching relationship.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including: a depth map processing device as in the second aspect.

[0007] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the depth map processing method of the first aspect are implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the depth map processing method of the first aspect are implemented.

[0009] In a sixth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the depth map processing method of the first aspect.

[0010] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the depth map processing method of the first aspect.

[0011] In the embodiment of the present application, multiple original depth maps of the target scene under multiple shooting angles are obtained, and the multiple shooting angles and the multiple original depth maps are obtained. Figure 1 One-to-one correspondence; according to the shooting parameters of multiple original depth maps, a pixel matching relationship between every two original depth maps in the multiple original depth maps is established; according to the preset correction model and the pixel matching relationship, the pixel depth values ​​of the multiple original depth maps are adjusted. Through the above-mentioned depth map processing method, after obtaining the corresponding multiple original depth maps of the target scene at multiple shooting angles, according to the shooting parameters of the multiple original depth maps, a matching relationship is established for the image pixels between every two original depth maps, and then the pixel depth values ​​of the multiple original depth maps are adjusted according to the established pixel matching relationship and the preset correction model. In this way, through the automated correction model and the pixel matching relationship between the original depth maps, the multiple original depth maps obtained at different shooting angles of the same target scene are corrected, thereby correcting the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the consistency of the distance measurement at each shooting angle, thereby ensuring the accuracy of the depth map display and the matching precision, and thus improving the accuracy of the point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of a depth map processing method according to an embodiment of the present invention;

[0013] Figure 2 A depth map processing result diagram provided in an embodiment of the present application;

[0014] Figure 3 A structural block diagram of a depth map processing device provided in an embodiment of the present application;

[0015] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0016] Figure 5One of the hardware schematic diagrams of the electronic device provided in the embodiment of the present application;

[0017] Figure 6 The second hardware schematic diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0019] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0020] The embodiment of the first aspect of the present application proposes a depth map processing method. The execution subject of the technical solution of the depth map processing method provided in the embodiment of the present application can be a depth map processing device, which can be determined according to actual use requirements and is not limited in the embodiment of the present application. In order to more clearly describe the depth map processing method provided in the embodiment of the present application, the execution subject of the depth map processing method is exemplarily described as a depth map processing device in the following method embodiment.

[0021] The depth map processing method provided in the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0022] like Figure 1 As shown, the embodiment of the present application provides a depth map processing method, which may include the following steps S102 to S106:

[0023] Step S102: Acquire multiple original depth maps of the target scene at multiple shooting angles.

[0024] The target scene is an application scene that needs to be scanned and measured. Specifically, the target scene can be an application scene that needs to be scanned and measured, such as an interior scene of a building or a statically placed object. In the actual application process, the target scene is scanned and measured by a depth camera to obtain a depth map with ranging information, and then the corresponding point cloud data is generated according to the obtained depth map.

[0025] The depth camera may be a camera with an infrared ranging function such as an RGB-D camera or a structured light depth camera. The user may select the type of the depth camera according to actual conditions, and no specific restrictions are made here.

[0026] Furthermore, the above-mentioned multiple original depth maps may be multiple depth maps obtained by the user through real-time shooting with a depth camera. The above-mentioned multiple original depth maps may be multiple depth maps in the local storage of the electronic device or shooting device (such as: local photo album or other image storage interval). The above-mentioned multiple original depth maps may also be multiple depth maps imported through an external storage device (for example: disk, USB flash drive). As for the specific source of the above-mentioned multiple original depth maps, the user can choose according to the actual situation, and no specific limitation is made here.

[0027] That is to say, the depth map processing method provided in the embodiment of the present application can process multiple original depth maps captured in real time during the process of shooting the original depth maps, or it can retrieve multiple original depth maps stored in a shooting device, electronic device or other storage device to process them after all the original depth maps are acquired.

[0028] Furthermore, the above-mentioned multiple shooting angles and multiple original depths Figure 1 One-to-one correspondence. The above-mentioned multiple shooting angles are related to the displacement parameters (e.g., moving direction, moving speed, etc.) and device parameters (e.g., shutter speed, etc.) of the depth camera during the shooting of the original depth map. During the shooting of the original depth map, the depth camera moves and shoots the target scene according to the preset displacement parameters and device parameters, thereby obtaining multiple original depth maps of the target scene at multiple shooting angles.

[0029] Step S104: establishing a pixel matching relationship between every two original depth maps in the plurality of original depth maps according to the shooting parameters of the plurality of original depth maps.

[0030] The shooting parameters include the displacement parameters of the depth camera during the original depth map shooting process (e.g., movement mode, movement direction, movement linear velocity, movement angular velocity, etc.) and device parameters (e.g., shutter speed, shooting delay, device built-in parameters, etc.). The shooting parameters are pre-stored in the gimbal or other storage device corresponding to the depth camera. In the actual application process, the shooting parameters can be directly called, so as to automatically process the multiple original depth maps obtained according to the shooting parameters.

[0031] Furthermore, the above-mentioned multiple original depth maps are obtained by moving the depth camera to shoot the target scene according to preset displacement parameters and device parameters, so as to shoot the same target scene from different shooting angles, thereby obtaining multiple depth maps of the target scene at multiple shooting angles.

[0032] Therefore, after obtaining multiple original depth maps of the target scene at multiple shooting angles, the multiple original depth maps can be matched in pairs according to the movement parameters such as the movement mode, movement direction, movement linear velocity, and movement angular velocity of the depth camera when shooting the multiple original depth maps, as well as the device parameters such as the shutter speed, shooting delay, and device built-in parameters. Specifically, the image pixels of every two original depth maps are matched according to the above-mentioned movement parameters and device parameters to determine the corresponding positions of the pixels in a certain original depth map in other original depth maps, thereby establishing a pixel matching relationship between every two original depth maps in the multiple original depth maps, so that the multiple original depth maps obtained can be corrected according to the pixel matching relationship later.

[0033] Among them, there are many moving modes of the above-mentioned depth camera, such as uniform rotation movement, uniform linear movement, uniform arc movement and other moving modes. For different moving modes, the corresponding pixel matching methods are also different. In the actual application process, the moving mode of the depth camera and the corresponding pixel matching method are pre-stored in the gimbal or other storage device corresponding to the depth camera. In the process of depth map processing, the above-mentioned pre-stored moving mode and pixel matching method can be directly called, so as to automatically establish a pixel matching relationship between each two original depth maps in multiple original depth maps according to the above-mentioned moving mode and pixel matching method, so as to subsequently perform correction processing on the obtained multiple original depth maps according to the pixel matching relationship.

[0034] Exemplarily, when the movement mode of the depth camera is rotational movement, the pixel matching relationship between each two original depth maps in the multiple original depth maps can be determined according to the rotation matrix of each original depth map and the built-in parameters of the depth camera. Specifically, the pixel matching relationship between each two original depth maps in the multiple original depth maps obtained by the depth camera during rotational movement can be determined according to the following formula:

[0035]

[0036] Among them, u i is an image pixel of a first original depth map among the multiple original depth maps, is the second original depth map in the multiple original depth maps corresponding to the above u i The image pixels that match is the rotation matrix of the first original depth map, is the rotation matrix of the second original depth map, is the transposed matrix of the rotation matrix of the second original depth map, and K is the built-in parameter of the depth camera. In this way, through the above formula, each image pixel between each two original depth maps in the multiple original depth maps captured by the rotation and movement of the depth camera is matched, so as to establish a pixel matching relationship between each two original depth maps in the multiple original depth maps, so as to subsequently perform correction processing on the obtained multiple original depth maps according to the pixel matching relationship.

[0037] Furthermore, when the movement mode of the above-mentioned depth camera is other movement modes, the motion matrix corresponding to each original depth map is obtained according to the specific movement mode of the depth camera, and the pixel matching formula corresponding to the movement mode can be obtained by replacing the rotation matrix of the original depth map in the above formula with the motion matrix. That is, for other movement modes, the pixel matching relationship between each two original depth maps in the multiple original depth maps obtained by the depth camera can be determined according to the following formula:

[0038]

[0039] Among them, u i is an image pixel of a first original depth map among the multiple original depth maps, is the second original depth map in the multiple original depth maps corresponding to the above u i The image pixels that match is the motion matrix of the first original depth map, is the motion matrix of the second original depth map, is the transposed matrix of the motion matrix of the second original depth map, and K is the built-in parameter of the depth camera.

[0040] Step S106: adjusting pixel depth values ​​of a plurality of original depth maps according to a preset correction model and a pixel matching relationship.

[0041] It can be understood that the depth map is an image used to obtain the depth information of objects in the target scene. The depth map is obtained through a depth camera to scan and measure the target scene to obtain the distance information of the target scene structure. Among them, each pixel value in the depth map represents the depth value between a certain point in the target scene and the depth camera at the time of shooting, that is, it represents the distance between a certain point in the target scene and the depth camera at the time of shooting, that is, it represents the distance from a certain point in the target scene to the plane where the depth camera at the time of shooting is located. Therefore, in the actual application process, the image pixel in the above-mentioned original depth map obtained can be adjusted to adjust the depth value represented by the pixel, thereby adjusting the depth information of the object in the target scene represented by the original depth map.

[0042] Specifically, after obtaining multiple original depth maps of the target scene at multiple shooting angles and establishing a pixel matching relationship between every two original depth maps in the multiple original depth maps, the image pixels of the multiple original depth maps are adjusted according to the above pixel matching relationship and a preset correction model, that is, the pixel depth values ​​of the multiple original depth maps are adjusted.

[0043] The preset correction model is a depth value correction model related to the pixel horizontal coordinates and pixel depth values ​​of the original depth map. By substituting the pixel horizontal coordinates and pixel depth values ​​of the image pixels of the original depth map into the depth value correction model, the depth values ​​of the image pixels are corrected to obtain the corrected pixel depth values ​​of the image pixels. The preset correction model can be specifically expressed by the following formula:

[0044]

[0045] Where b is the baseline length of the depth camera, f is the focal length of the depth camera, p and q are the model parameters of the preset correction model, x is the horizontal coordinate of the image pixel in the original depth map, and d is the horizontal coordinate of the image pixel in the original depth map. raw is the original depth value of the image pixel corresponding to x, d new For d raw The corrected depth value obtained after correction.

[0046] In this way, when the pixel depth value of a certain original depth map is adjusted by the preset correction model, the original depth map is traversed to obtain the pixel horizontal coordinates of all image pixels in the image and the corresponding pixel depth values, and then the obtained pixel horizontal coordinates and pixel depth values ​​of each image pixel are substituted into the above-mentioned preset correction model to correct the depth value of each image pixel, thereby obtaining the corrected pixel depth value of each image pixel, ensuring the consistency of the ranging information represented by the original depth map of the target scene at each shooting angle, correcting the measurement error caused by the ambient temperature to multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0047] Through the above-mentioned depth map processing method provided by the embodiment of the present application, after obtaining the corresponding multiple original depth maps of the target scene at multiple shooting angles, a matching relationship is established between the image pixels of each two original depth maps according to the shooting parameters of the multiple original depth maps, and then the pixel depth values ​​of the multiple original depth maps are adjusted according to the established pixel matching relationship and the preset correction model. In this way, through the automated correction model and the pixel matching relationship between the original depth maps, the multiple original depth maps obtained at different shooting angles of the same target scene are corrected, thereby correcting the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the consistency of the distance measurement at each shooting angle, thereby ensuring the accuracy of the depth map display and the matching precision, and thus improving the accuracy of the point cloud data.

[0048] In the embodiment of the present application, the above step S104 may specifically include the following steps S104a and S104b:

[0049] Step S104a: Acquire the displacement parameters and device parameters of the shooting device.

[0050] The above-mentioned shooting device is a depth camera with an infrared ranging function, and the shooting device can specifically be a depth camera such as an RGB-D camera, a structured light depth camera, etc. As for the specific type of the above-mentioned shooting device, the user can select it according to the actual situation, and no specific limitation is made here.

[0051] Furthermore, the above-mentioned displacement parameters include movement information such as the movement mode, movement direction, movement linear velocity, and movement angular velocity of the shooting device during the shooting process, and the above-mentioned device parameters include device information such as the shutter speed, shooting delay, and built-in parameters of the device.

[0052] In the actual application process, the above shooting parameters are pre-stored in the pan / tilt or other storage device corresponding to the shooting device, and the shooting device moves and shoots the target scene according to the above preset displacement parameters and device parameters, so as to shoot the same target scene from different shooting angles, thereby obtaining multiple original depth maps of the target scene at multiple shooting angles. On this basis, after obtaining multiple original depth maps of the target scene at multiple shooting angles, the above displacement parameters and device parameters can be directly retrieved from the pre-stored interval, so as to automatically process the obtained multiple original depth maps according to the above displacement parameters and device parameters.

[0053] Step S104b: Determine the pixel position of the photographed subject in the plurality of original depth maps according to the displacement parameters and the device parameters, and establish a pixel matching relationship according to the pixel position.

[0054] Specifically, after obtaining multiple original depth maps of the target scene at multiple shooting angles, a pixel matching relationship between every two original depth maps can be established based on movement parameters such as the movement mode, movement direction, movement linear velocity, and movement angular velocity of the shooting device when shooting the multiple original depth maps, as well as device parameters such as shutter speed, shooting delay, and device built-in parameters.

[0055] Specifically, the pixel position of the photographed subject in the target scene in each original depth map is determined according to the above-mentioned movement parameters and device parameters, and then the image pixels of every two original depth maps are matched according to the pixel position of the photographed subject in each original depth map, and the corresponding position of the pixel point in a certain original depth map in the other original depth maps is determined, so as to establish a pixel matching relationship between every two original depth maps in the multiple original depth maps, so that the multiple original depth maps obtained can be corrected subsequently according to the pixel matching relationship.

[0056] Among them, there are many moving modes of the above-mentioned shooting device, such as uniform rotation movement, uniform linear movement, uniform arc movement and other moving modes. For different moving modes, the corresponding pixel matching methods are also different. In the actual application process, the moving mode of the shooting device and the corresponding pixel matching method are pre-stored in the gimbal or other storage device corresponding to the shooting device. During the depth map processing process, the moving mode of the above-mentioned shooting device and the corresponding pixel matching method can be directly called from the preset storage interval, so as to automatically establish a pixel matching relationship between each two original depth maps in the multiple original depth maps according to the above-mentioned moving mode and pixel matching method, so as to subsequently perform correction processing on the multiple original depth maps obtained according to the pixel matching relationship.

[0057] Exemplarily, when the movement mode of the above-mentioned shooting device is rotational movement, the pixel matching relationship between each two original depth maps in the above-mentioned multiple original depth maps can be determined according to the rotation matrix of each original depth map and the built-in parameters of the shooting device. Specifically, the pixel matching relationship between each two original depth maps in the multiple original depth maps obtained by the shooting device during rotational movement can be determined according to the following formula:

[0058]

[0059] Among them, u i is an image pixel of a first original depth map among the multiple original depth maps, is the second original depth map in the multiple original depth maps corresponding to the above u i The image pixels that match is the rotation matrix of the first original depth map, is the rotation matrix of the second original depth map, is the transposed matrix of the rotation matrix of the second original depth map, and K is a built-in parameter of the shooting device. In this way, through the above formula, each image pixel between each two original depth maps in the multiple original depth maps captured by the shooting device through rotation and movement is matched, so as to establish a pixel matching relationship between each two original depth maps in the multiple original depth maps, so as to subsequently perform correction processing on the obtained multiple original depth maps according to the pixel matching relationship.

[0060] Furthermore, when the movement mode of the above-mentioned depth camera is other movement modes, the motion matrix corresponding to each original depth map is obtained according to the specific movement mode of the depth camera, and the pixel matching formula corresponding to the movement mode can be obtained by replacing the rotation matrix of the original depth map in the above formula with the motion matrix. That is, for other movement modes, the pixel matching relationship between each two original depth maps in the multiple original depth maps obtained by the depth camera can be determined according to the following formula:

[0061]

[0062] Among them, u i is an image pixel of a first original depth map among the multiple original depth maps, is the second original depth map in the multiple original depth maps corresponding to the above u i The image pixels that match is the motion matrix of the first original depth map, is the motion matrix of the second original depth map, is the transposed matrix of the motion matrix of the second original depth map, and K is the built-in parameter of the depth camera.

[0063] In the above-mentioned embodiment provided by the present application, when establishing a pixel matching relationship between every two original depth maps in a plurality of original depth maps, the displacement parameters and device parameters of the shooting device are obtained, and then the pixel positions of the shooting subject in the target scene in the plurality of original depth maps are determined according to the obtained displacement parameters and device parameters, and the pixel matching relationship between every two original depth maps is established according to the determined pixel positions. In this way, the pixel matching relationship between every two original depth maps is established according to the displacement parameters and device parameters of the shooting device, which ensures the accuracy of establishing the pixel matching relationship between the original depth maps, thereby ensuring the accuracy of depth map processing, correcting the measurement errors caused by the ambient temperature to the plurality of original depth maps, and ensuring the consistency of the ranging information represented by the original depth maps of the target scene at each shooting angle.

[0064] In the embodiment of the present application, before adjusting the pixel depth values ​​of the plurality of original depth maps, the depth map processing method further includes the following step S105:

[0065] Step S105: Optimizing model parameters of a preset correction model according to a plurality of original depth maps and pixel matching relationships.

[0066] The multiple original depth maps are obtained by shooting the target scene by moving the shooting device according to the preset displacement parameters and device parameters, so as to shoot the same target scene from different shooting angles, thereby obtaining multiple depth maps of the target scene at multiple shooting angles. Figure 1 One to one correspondence.

[0067] Furthermore, the above-mentioned pixel matching relationship is to determine the pixel position of the photographed subject in the target scene in each original depth map according to the above-mentioned movement parameters and device parameters, and then match the image pixels of every two original depth maps according to the pixel position of the photographed subject in each original depth map, and determine the corresponding position of the pixel point in a certain original depth map in other original depth maps, thereby establishing a pixel matching relationship between every two original depth maps in multiple original depth maps.

[0068] Specifically, when the image pixels of multiple original depth maps are adjusted according to the above-mentioned pixel matching relationship and the preset correction model, that is, when the pixel depth values ​​of multiple original depth maps are adjusted, due to the differences in the movement parameters of the shooting device, the device parameters and the pixel matching relationship between the original depth maps when shooting the original depth maps, the above-mentioned preset correction model has different processing adaptability for the original depth maps, thereby affecting the accuracy of the depth map processing results.

[0069] Therefore, in an embodiment of the present application, before adjusting the image pixels of multiple original depth maps through a preset correction model, that is, before adjusting the pixel depth values ​​of multiple original depth maps, the model parameters of the preset correction model are first iteratively optimized according to the above-mentioned multiple original depth maps and pixel matching relationships to obtain optimized target model parameters, and then the target model parameters are substituted into the original preset correction model to obtain an optimized target correction model, and then the image pixels of multiple original depth maps are adjusted through the target correction model, that is, the pixel depth values ​​of multiple original depth maps are adjusted. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0070] Specifically, when optimizing the model parameters of the preset correction model, the pixel matching relationships between the above-mentioned multiple original depth maps are traversed. For each pair of pixel matching relationships, the preset correction model is applied to the image pixels in the two original depth maps corresponding to the pixel matching relationship, and two corresponding corrected depth values ​​are obtained for each pair of matching pixels. On this basis, for each pair of matching pixels, a sub-optimization function with model parameters as variables is established according to the two corresponding corrected depth values, and then the final target optimization function is established according to the sum of all sub-optimization functions established for each pair of matching pixels in the above-mentioned multiple original depth maps, and then the established target optimization function is optimized according to the preset optimization algorithm to obtain a set of optimized target model parameters.

[0071] For each pair of matching pixels, the two corresponding corrected depth values ​​can be obtained by the following formula:

[0072]

[0073]

[0074] Where b is the baseline length of the shooting device, f is the focal length of the depth camera, p and q are the model parameters of the preset calibration model, x1 is the horizontal coordinate of the first image pixel in a pair of matching pixels, d1 is the original depth value of the image pixel corresponding to x1, and d correct (p, q, d1, x1) is the corrected depth value corresponding to d1, x2 is the horizontal coordinate of the second image pixel in a pair of matching pixels, d2 is the original depth value of the image pixel corresponding to x2, and d correct (p, q, d2, x2) is the corrected depth value corresponding to d2.

[0075] On this basis, for each pair of matching pixels, the corresponding sub-optimization function can be expressed by the following formula:

[0076]

[0077] Among them, d correct (p, q, d1, x1) is the corrected depth value corresponding to the first image pixel in a pair of matching pixels, d correct (p, q, d2, x2) is the corrected depth value corresponding to the second image pixel in a pair of matching pixels.

[0078] Furthermore, the above objective optimization function can be specifically expressed by the following formula:

[0079]

[0080] Among them, d correct (p, q, d1, x1) is the corrected depth value corresponding to the first image pixel in each pair of matching pixels, d correct (p, q, d2, x2) is the corrected depth value corresponding to the second image pixel in each pair of matching pixels.

[0081] In addition, to ensure the accuracy of model parameter optimization, in actual application, a target optimization function can be established according to the above method based on every two original depth maps in multiple original depth maps and the corresponding pixel matching relationship, and then the model parameters can be iteratively optimized to determine the optimal model parameters.

[0082] In the above embodiment provided by the present application, before adjusting the pixel depth values ​​of multiple original depth maps by using a preset correction model, the model parameters of the preset correction model are first iteratively optimized according to the above multiple original depth maps and pixel matching relationships to obtain optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0083] In the embodiment of the present application, the above step S105 may further include the following steps S105a to S105d:

[0084] Step S105a: Divide the multiple pixels in the multiple original depth maps into multiple groups according to the pixel matching relationship.

[0085] Specifically, the pixel matching relationships among the multiple original depth maps are traversed, and according to the pixel matching relationship between every two original depth maps among the multiple original depth maps, multiple pixel points in the multiple original depth maps are grouped to obtain multiple groups of pixel points.

[0086] Each group of pixels includes two pixels that match each other and are located in different original depth maps.

[0087] Step S105b: Substitute the depth value of each group of pixel points in the multiple groups of pixel points into a preset correction model to obtain multiple groups of target depth values.

[0088] Specifically, for each of the above-mentioned multiple groups of pixels, the preset correction model is applied to the two image pixels corresponding to the group of pixels, respectively, so as to obtain a corrected set of target depth values ​​corresponding to the group of pixels. On this basis, for each group of pixels, a corresponding set of target depth values ​​is obtained through the preset correction model, thereby obtaining multiple sets of target depth values. Among them, there is a one-to-one correspondence between the multiple sets of target depth values ​​and the multiple sets of pixels.

[0089] For each group of pixels, a corresponding set of target depth values ​​can be obtained by the following formula:

[0090]

[0091]

[0092] Where b is the baseline length of the shooting device, f is the focal length of the depth camera, p and q are the model parameters of the preset calibration model, x1 is the horizontal coordinate of the first pixel in a group of pixels, d1 is the original depth value of the pixel corresponding to x1, and d correct (p, q, d1, x1) is the target depth value corresponding to d1, x2 is the horizontal coordinate of the second pixel in a group of pixels, d2 is the original depth value of the pixel corresponding to x2, d correct (p,q,d2,x2) is the target depth value corresponding to d2.

[0093] Step S105c: establishing a target optimization function according to multiple groups of target depth values.

[0094] Specifically, after the preset correction model is applied to the above-mentioned multiple groups of pixels to obtain the corresponding multiple groups of target depth values, for each group of matched pixels, a sub-optimization function with model parameters as variables is established according to its corresponding group of target depth values, and then the final target optimization function is established according to the sum of multiple sub-optimization functions corresponding to the multiple groups of pixels. The target optimization function is a function with model parameters of the preset correction model as variables.

[0095] Among them, for each group of pixels, the corresponding sub-optimization function can be expressed by the following formula:

[0096]

[0097] Among them, d correct(p, q, d1, x1) is the target depth value corresponding to the first pixel in a group of pixels, d correct (p, q, d2, x2) is the target depth value corresponding to the second pixel in a group of pixels.

[0098] On this basis, the above objective optimization function can be specifically expressed by the following formula:

[0099]

[0100] Among them, d correct (p, q, d1, x1) is the target depth value corresponding to the first pixel in each group of pixels, d correct (p, q, d2, x2) is the target depth value corresponding to the second pixel in each group of pixels.

[0101] Step S105d: Optimize the target optimization function according to a preset optimization algorithm to obtain optimized target model parameters.

[0102] Specifically, after the preset correction model is applied to the above-mentioned multiple groups of pixel points respectively to obtain the corresponding multiple groups of target depth values, and the target optimization function is established according to the multiple groups of target depth values, the established target optimization function is optimized according to the preset optimization algorithm to obtain the optimized target model parameters, and then the optimized target correction model is determined according to the target model parameters.

[0103] Among them, the above-mentioned preset optimization algorithm can be specifically LM (Levenberg-Marquardt, Levenberg-Marquardt algorithm, and the LM algorithm is used to achieve rapid optimization of model parameters. In the actual application process, the above-mentioned target optimization function can also be optimized by other optimization algorithms, so as to optimize the model parameters of the preset correction model. For the specific optimization algorithm of the target optimization function, the user can select it according to the actual situation, and no specific restriction is made here.

[0104] In the above-mentioned embodiment provided by the present application, when the model parameters of the preset correction model are iteratively optimized according to the above-mentioned multiple original depth maps and pixel matching relationships, multiple pixel points in the multiple original depth maps are divided into multiple groups according to the pixel matching relationships, and then the depth values ​​of each group of pixel points in the multiple groups of pixel points are substituted into the preset correction model to obtain multiple groups of target depth values, and a target optimization function is established according to the obtained multiple groups of target depth values, and then the target optimization function is optimized by a preset optimization algorithm to obtain the optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0105] In the embodiment of the present application, the above adjustment of the pixel depth values ​​of the plurality of original depth maps may specifically include the following step S106a:

[0106] Step S106a: Substituting the target model parameters into the preset correction model to obtain the target correction model, and adjusting the pixel depth values ​​of the plurality of original depth maps according to the target correction model.

[0107] Specifically, after iteratively optimizing the model parameters of the preset correction model according to the above-mentioned multiple original depth maps and pixel matching relationships to obtain optimized target model parameters, the target model parameters are substituted into the original preset correction model to obtain the optimized target correction model, and then the image pixels of the multiple original depth maps are adjusted through the target correction model, that is, the pixel depth values ​​of the multiple original depth maps are adjusted.

[0108] The target correction model is a depth value correction model related to the pixel horizontal coordinates and pixel depth values ​​of the original depth map. By substituting the pixel horizontal coordinates and pixel depth values ​​of the image pixels of the original depth map into the target correction model, the depth values ​​of the image pixels are corrected to obtain the corrected pixel depth values ​​of the image pixels. The target correction model can be specifically expressed by the following formula:

[0109]

[0110] Where b is the baseline length of the camera, f is the focal length of the camera, p0 and q0 are the target model parameters, x is the horizontal coordinate of the image pixel in the original depth map, and d ,aw is the original depth value of the image pixel corresponding to x, d new For d ,aw The corrected depth value obtained after correction.

[0111] In this way, when the pixel depth value of a certain original depth map is adjusted by the target correction model, the original depth map is traversed to obtain the pixel horizontal coordinates of all image pixels in the image and the corresponding pixel depth values, and then the obtained pixel horizontal coordinates and pixel depth values ​​of each image pixel are substituted into the above-mentioned target correction model to correct the depth value of each image pixel, thereby obtaining the corrected pixel depth value of each image pixel, ensuring the consistency of the ranging information represented by the original depth map of the target scene at each shooting angle, correcting the measurement error caused by the ambient temperature to multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0112] In the above embodiment provided by the present application, the target model parameters are substituted into the preset correction model to obtain the target correction model, and then the pixel depth values ​​of multiple original depth maps are adjusted according to the target correction model. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement error caused by the ambient temperature to the multiple original depth maps, ensuring the consistency of the distance measurement of each shooting angle, ensuring the accuracy of the depth map display, thereby ensuring the accuracy of the depth map display and the matching accuracy, and thus improving the accuracy of the point cloud data.

[0113] In the embodiment of the present application, after the above step S106, the above depth map processing method may further include the following step S107:

[0114] Step S107: generating a plurality of point cloud data according to the adjusted plurality of original depth maps through conversion.

[0115] Among them, multiple original depth maps and multiple point cloud data correspond one to one.

[0116] Among them, the point cloud is a massive set of points that express the spatial distribution and surface characteristics of the target in the same spatial reference system. After obtaining the spatial coordinates of each sampling point on the surface of the object, the resulting set of points is called a "point cloud".

[0117] It is understandable that the depth map and the point cloud are two different ways of expressing the depth information of the objects in the target scene, and the two can be converted into each other. Therefore, after the original depth map obtained is processed in the above manner to correct the measurement error caused by the ambient temperature to the original depth map, the corrected original depth map can be converted into point cloud data according to the conversion relationship between the depth map and the point cloud, so as to express the depth information of the objects in the target scene through the point cloud data.

[0118] Specifically, the conversion of a depth map into a point cloud is essentially the conversion of coordinate systems. Based on the correspondence between coordinate systems, the spatial coordinate system corresponding to the point cloud is converted and generated according to the image coordinate system of the depth map and the pixel depth value, thereby generating the corresponding point cloud data.

[0119] In the above embodiment provided by the present application, after the original depth map obtained is processed in the above manner to correct the measurement error caused by the ambient temperature on the original depth map, the corrected original depth map is converted into point cloud data according to the conversion relationship between the depth map and the point cloud. In this way, the measurement error caused by the ambient temperature on the point cloud data is corrected to ensure the consistency of the distance measurement of each shooting angle, thereby solving the problem that the point cloud cannot be aligned at the visual seam and produces obvious stratification and noise, improving the accuracy of the point cloud data, and thus ensuring the accuracy of the subsequent point cloud stitching operation.

[0120] In the related art, when a target scene is photographed by a depth camera equipped with an infrared sensor module to obtain a three-dimensional point cloud model to achieve scanning and ranging of the target scene structure, the infrared sensor module will be affected by the ambient temperature.

[0121] Specifically, in any target scene that deviates from the calibration temperature, the distance measurement result will be deflected around a certain spatial axis. The above deflection phenomenon is displayed as different measurement noise distributions in the depth map, and on the point cloud it is manifested as a non-rigid rotation of the point cloud. The above deflection phenomenon is the temperature deflection error, which leads to inconsistent ranging information obtained when measuring the same position at different viewing angles. Moreover, at different temperatures, the temperature deflection error also manifests differently. In this way, when a single-point point cloud is generated, the collected point cloud cannot be aligned at the visual seam, resulting in obvious stratification and noise, affecting subsequent point cloud operations such as object pose estimation and stitching.

[0122] Therefore, in the depth map processing method proposed in the embodiment of the present application, the original depth measurement data of a certain point at different viewing angles, i.e., the original depth map, is obtained, and then the obtained original depth map is traversed, and the matching relationship between the image pixels of the current original depth map and the image pixels of other original depth maps is established according to the pan / tilt rotation angle and calibration parameters of the shooting device. On this basis, the matching relationship established above is traversed, and for each pair of matching relationships, the preset correction model is applied to the current depth measurement data, i.e., the original depth value, to obtain the corrected target depth value, and then a target optimization function is established according to the target depth value, and the target optimization function is optimized by algorithms such as LM to obtain the optimal model parameters (i.e., target model parameters). Based on this, the obtained target model parameters are substituted into the preset correction model to obtain the target correction model, and then the image pixels of all the original depth maps are traversed to obtain the horizontal coordinates of each pixel and the corresponding original depth value, and the original depth value of each image pixel is corrected by the target correction model to obtain the corresponding target depth value, and then the corrected target depth map is obtained, and the point cloud data is generated by the conversion of the target depth map. In this way, the original depth measurement data is corrected through automated correction, so that the distance information obtained by measuring the same position at different viewing angles is consistent, and the temperature deflection error caused by the ambient temperature is corrected, so that the single-point site cloud can be aligned and displayed at the visual seam.

[0123] For example, Figure 2 The figure shows the display changes of point cloud data before and after correction. Figure 2 As shown in (a) of Figure 1, due to the influence of ambient temperature, point cloud overlap occurs in the box, that is, the point clouds cannot be aligned at the visual seams. On this basis, after the original depth measurement data obtained is corrected by the above method, as shown in Figure 2As shown in (b), the above-mentioned point cloud duplication phenomenon disappears, and the point clouds can be aligned and displayed at the visual seams.

[0124] In addition, the above-mentioned depth map processing method proposed in the embodiment of the present application can be completed automatically without the need for human intervention, and the processing method is adaptable to different cameras, different temperatures, different acquisition schemes and different usage scenarios, corrects the point cloud measurement error, and greatly reduces the difficulty of subsequent algorithm processing.

[0125] The depth map processing method provided in the embodiment of the first aspect of the present application may be executed by a depth map processing device. In the embodiment of the present application, the depth map processing device performing the above-mentioned depth map processing method is taken as an example to illustrate the depth map processing device provided in the embodiment of the second aspect of the present application.

[0126] like Figure 3 As shown, an embodiment of the present application provides a depth map processing device 300, which may include the following acquisition unit 302 and processing unit 304.

[0127] The acquisition unit 302 is used to acquire multiple original depth maps of the target scene at multiple shooting angles, multiple shooting angles and multiple original depth maps. Figure 1 One to one correspondence;

[0128] A processing unit 304 is used to establish a pixel matching relationship between every two original depth maps in the multiple original depth maps according to the shooting parameters of the multiple original depth maps;

[0129] The processing unit 304 is further configured to adjust the pixel depth values ​​of the plurality of original depth maps according to the preset correction model and the pixel matching relationship.

[0130] In the embodiment of the present application, after the acquisition unit 302 acquires the corresponding multiple original depth maps of the target scene at multiple shooting angles, the processing unit 304 establishes a matching relationship between the image pixels of each two original depth maps according to the shooting parameters of the multiple original depth maps, and then adjusts the pixel depth values ​​of the multiple original depth maps according to the established pixel matching relationship and the preset correction model by the processing unit 304. In this way, through the automated correction model and the pixel matching relationship between the original depth maps, the multiple original depth maps obtained at different shooting angles of the same target scene are corrected, thereby correcting the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the consistency of the distance measurement at each shooting angle, thereby ensuring the accuracy of the depth map display and the matching precision, and thus improving the accuracy of the point cloud data.

[0131] In the embodiment of the present application, the processing unit 304 is specifically used to: obtain displacement parameters and device parameters of the shooting device; determine the pixel position of the shooting subject in multiple original depth maps based on the displacement parameters and device parameters; and establish a pixel matching relationship based on the pixel position.

[0132] In the above-mentioned embodiment provided by the present application, when the pixel matching relationship between each two original depth maps in the multiple original depth maps is established by the processing unit 304, the displacement parameters and device parameters of the shooting device are obtained, and then the pixel positions of the shooting subjects in the target scene in the multiple original depth maps are determined according to the obtained displacement parameters and device parameters, and the pixel matching relationship between each two original depth maps is established according to the determined pixel positions. In this way, the pixel matching relationship between each two original depth maps is established according to the displacement parameters and device parameters of the shooting device, which ensures the accuracy of the establishment of the pixel matching relationship between the original depth maps, thereby ensuring the accuracy of the depth map processing, correcting the measurement errors caused by the ambient temperature to the multiple original depth maps, and ensuring the consistency of the ranging information represented by the original depth maps of the target scene at each shooting angle.

[0133] In the embodiment of the present application, the processing unit 304 may also be used to optimize the model parameters of the preset correction model according to the multiple original depth maps and pixel matching relationships.

[0134] In the above embodiment provided by the present application, before the processing unit 304 adjusts the pixel depth values ​​of multiple original depth maps through the preset correction model, the processing unit 304 will also iteratively optimize the model parameters of the preset correction model according to the above multiple original depth maps and the pixel matching relationship to obtain the optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0135] In an embodiment of the present application, the processing unit 304 can be specifically used to: divide multiple pixel points in multiple original depth maps into multiple groups according to a pixel matching relationship, wherein each group of pixel points includes two pixel points that match each other and are located in different original depth maps; substitute the depth value of each group of pixel points in the multiple groups of pixel points into a preset correction model to obtain multiple groups of target depth values; establish a target optimization function based on the multiple groups of target depth values; optimize the target optimization function according to a preset optimization algorithm to obtain optimized target model parameters.

[0136] In the above-mentioned embodiment provided by the present application, when the model parameters of the preset correction model are iteratively optimized by the processing unit 304 according to the above-mentioned multiple original depth maps and pixel matching relationships, multiple pixel points in the multiple original depth maps are divided into multiple groups according to the pixel matching relationships, and then the depth values ​​of each group of pixel points in the multiple groups of pixel points are substituted into the preset correction model to obtain multiple groups of target depth values, and a target optimization function is established according to the obtained multiple groups of target depth values, and then the target optimization function is optimized by a preset optimization algorithm to obtain optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0137] In the embodiment of the present application, the processing unit 304 may be specifically configured to: substitute the target model parameters into a preset correction model to obtain a target correction model; and adjust the pixel depth values ​​of the plurality of original depth maps according to the target correction model.

[0138] In the above embodiment provided by the present application, the target model parameters are substituted into the preset correction model by the processing unit 304 to obtain the target correction model, and then the pixel depth values ​​of the multiple original depth maps are adjusted according to the target correction model. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing result, reducing the measurement error caused by the ambient temperature to the multiple original depth maps, ensuring the consistency of the distance measurement of each shooting angle, ensuring the accuracy of the depth map display, thereby ensuring the accuracy of the depth map display and the matching accuracy, and thus improving the accuracy of the point cloud data.

[0139] In the embodiment of the present application, the processing unit 304 may also be used to: generate a plurality of point cloud data according to the adjusted plurality of original depth maps, wherein the plurality of original depth maps and the plurality of point cloud data correspond one to one.

[0140] In the above embodiment provided by the present application, after the original depth map obtained is processed in the above manner to correct the measurement error caused by the ambient temperature on the original depth map, the corrected original depth map can also be converted into point cloud data according to the conversion relationship between the depth map and the point cloud by the processing unit 304. In this way, the measurement error caused by the ambient temperature on the point cloud data is corrected to ensure the consistency of the distance measurement of each shooting angle, thereby solving the problem that the point cloud cannot be aligned at the visual seam and produces obvious stratification and noise, improving the accuracy of the point cloud data, and thus ensuring the accuracy of the subsequent point cloud splicing operation.

[0141] The depth map processing device 300 in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0142] The depth map processing device 300 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0143] The depth map processing device 300 provided in the second aspect of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0144] Alternatively, if Figure 4 As shown, the embodiment of the present application further provides an electronic device 400, including the depth map processing device 300 as described in the second aspect. Therefore, the electronic device 400 provided by the embodiment of the present invention can achieve all the technical effects of the depth map processing device 300 of the second aspect, and will not be described here to avoid repetition.

[0145] Alternatively, if Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 502 and a memory 504, wherein the memory 504 stores programs or instructions that can be executed on the processor 502, and when the program or instructions are executed by the processor 502, the various steps of the depth map processing method embodiment of the above-mentioned first aspect are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0146] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0147] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0148] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0149] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0150] The electronic device 600 of the embodiment of the present application can be used to implement the various steps of the embodiment of the depth map processing method of the first aspect mentioned above.

[0151] The processor 610 is used to obtain multiple original depth maps of the target scene at multiple shooting angles, multiple shooting angles and multiple original depth maps. Figure 1 One to one correspondence.

[0152] The processor 610 is further configured to establish a pixel matching relationship between every two original depth maps in the multiple original depth maps according to the shooting parameters of the multiple original depth maps.

[0153] The processor 610 is further configured to adjust pixel depth values ​​of the plurality of original depth maps according to a preset correction model and a pixel matching relationship.

[0154] In the embodiment of the present application, after the electronic device 600 obtains the corresponding multiple original depth maps of the target scene at multiple shooting angles through the processor 610, the processor 610 establishes a matching relationship between the image pixels of each two original depth maps according to the shooting parameters of the multiple original depth maps, and then adjusts the pixel depth values ​​of the multiple original depth maps according to the established pixel matching relationship and the preset correction model through the processor 610. In this way, through the automated correction model and the pixel matching relationship between the original depth maps, the multiple original depth maps obtained at different shooting angles of the same target scene are corrected, thereby correcting the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the consistency of the distance measurement at each shooting angle, thereby ensuring the accuracy of the depth map display and the matching accuracy, and thus improving the accuracy of the point cloud data.

[0155] Optionally, the processor 610 is specifically used to: obtain displacement parameters and device parameters of the shooting device; determine the pixel position of the shooting subject in multiple original depth maps according to the displacement parameters and the device parameters; and establish a pixel matching relationship according to the pixel position.

[0156] In the above-mentioned embodiment provided by the present application, when the pixel matching relationship between each two original depth maps in the multiple original depth maps is established by the processor 610, the displacement parameters and device parameters of the shooting device are obtained, and then the pixel positions of the shooting subjects in the target scene in the multiple original depth maps are determined according to the obtained displacement parameters and device parameters, and the pixel matching relationship between each two original depth maps is established according to the determined pixel positions. In this way, the pixel matching relationship between each two original depth maps is established according to the displacement parameters and device parameters of the shooting device, which ensures the accuracy of the establishment of the pixel matching relationship between the original depth maps, thereby ensuring the accuracy of the depth map processing, correcting the measurement errors caused by the ambient temperature to the multiple original depth maps, and ensuring the consistency of the ranging information represented by the original depth maps of the target scene at each shooting angle.

[0157] Optionally, the processor 610 may also be configured to: optimize model parameters of a preset correction model according to a plurality of original depth maps and pixel matching relationships.

[0158] In the above embodiment provided by the present application, before the processor 610 adjusts the pixel depth values ​​of multiple original depth maps through the preset correction model, the processor 610 will also iteratively optimize the model parameters of the preset correction model according to the above multiple original depth maps and the pixel matching relationship to obtain the optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0159] Optionally, the processor 610 can be specifically used to: divide multiple pixel points in multiple original depth maps into multiple groups according to a pixel matching relationship, wherein each group of pixel points includes two pixel points that match each other and are located in different original depth maps; substitute the depth value of each group of pixel points in the multiple groups of pixel points into a preset correction model to obtain multiple groups of target depth values; establish a target optimization function based on the multiple groups of target depth values; optimize the target optimization function according to a preset optimization algorithm to obtain optimized target model parameters.

[0160] In the above-mentioned embodiment provided by the present application, when the model parameters of the preset correction model are iteratively optimized by the processor 610 according to the above-mentioned multiple original depth maps and pixel matching relationships, multiple pixel points in the multiple original depth maps are divided into multiple groups according to the pixel matching relationships, and then the depth values ​​of each group of pixel points in the multiple groups of pixel points are substituted into the preset correction model to obtain multiple groups of target depth values, and a target optimization function is established according to the obtained multiple groups of target depth values, and then the target optimization function is optimized by a preset optimization algorithm to obtain the optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0161] Optionally, the processor 610 may be specifically configured to: substitute the target model parameters into a preset correction model to obtain a target correction model; and adjust pixel depth values ​​of a plurality of original depth maps according to the target correction model.

[0162] In the above-mentioned embodiment provided by the present application, when the model parameters of the preset correction model are iteratively optimized by the processor 610 according to the above-mentioned multiple original depth maps and pixel matching relationships, multiple pixel points in the multiple original depth maps are divided into multiple groups according to the pixel matching relationships, and then the depth values ​​of each group of pixel points in the multiple groups of pixel points are substituted into the preset correction model to obtain multiple groups of target depth values, and a target optimization function is established according to the obtained multiple groups of target depth values, and then the target optimization function is optimized by a preset optimization algorithm to obtain the optimized target model parameters. In this way, the processing adaptability of the correction model to the original depth map is improved, thereby improving the accuracy of the depth map processing results, reducing the measurement errors caused by the ambient temperature to the multiple original depth maps, ensuring the accuracy of the depth map display, and thus improving the accuracy of the point cloud data.

[0163] Optionally, the processor 610 may be further configured to: generate a plurality of point cloud data according to the adjusted plurality of original depth maps, wherein the plurality of original depth maps and the plurality of point cloud data correspond one to one.

[0164] In the above embodiment provided by the present application, after the original depth map obtained is processed in the above manner to correct the measurement error caused by the ambient temperature on the original depth map, the corrected original depth map can also be converted into point cloud data according to the conversion relationship between the depth map and the point cloud by the processor 610. In this way, the measurement error caused by the ambient temperature on the point cloud data is corrected to ensure the consistency of the distance measurement of each shooting angle, thereby solving the problem that the point cloud cannot be aligned at the visual seam and produces obvious stratification and noise, improving the accuracy of the point cloud data, and thus ensuring the accuracy of the subsequent point cloud splicing operation.

[0165] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0166] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0167] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0168] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the depth map processing method embodiment of the first aspect described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0169] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0170] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the depth map processing method embodiment of the first aspect above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0171] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0172] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the depth map processing method embodiment of the first aspect mentioned above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0173] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0175] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A depth map processing method, characterized in that: include: Acquire multiple original depth maps of the target scene at multiple shooting angles, wherein the multiple shooting angles correspond to the multiple original depth maps one by one; Establishing a pixel matching relationship between every two original depth maps in the plurality of original depth maps according to shooting parameters of the plurality of original depth maps; Adjusting pixel depth values ​​of the plurality of original depth maps according to a preset correction model and the pixel matching relationship; Before adjusting the pixel depth values ​​of the plurality of original depth maps, the processing method further includes: Optimizing the model parameters of the preset correction model according to the multiple original depth maps and the pixel matching relationship; The optimizing the model parameters of the preset correction model according to the multiple original depth maps and the pixel matching relationship specifically includes: Dividing the plurality of pixels in the plurality of original depth maps into a plurality of groups according to the pixel matching relationship, wherein each group of pixels includes two pixels in different original depth maps that match each other; Substituting the depth value of each group of pixel points in the multiple groups of pixel points into the preset correction model to obtain multiple groups of target depth values; Establishing a target optimization function according to the multiple sets of target depth values; Optimizing the target optimization function according to a preset optimization algorithm to obtain optimized target model parameters; The preset correction model is expressed by the following formula: Where b is the baseline length of the depth camera, f is the focal length of the depth camera, p and q are the model parameters of the preset correction model, x is the horizontal coordinate of the image pixel in the original depth map, and d is the horizontal coordinate of the image pixel in the original depth map. raw is the original depth value of the image pixel corresponding to x, d new For d raw The corrected depth value obtained after the correction; The pixel matching relationships between the multiple original depth maps are traversed, and for each pair of pixel matching relationships, the preset correction model is respectively applied to the image pixels in the two original depth maps corresponding to the pixel matching relationship, and two corresponding corrected depth values ​​are obtained for each pair of matching pixels. For each pair of matching pixels, a sub-optimization function with model parameters as variables is established according to the difference between the corresponding two corrected depth values, and the sum of all sub-optimization functions is established according to each pair of matching pixels in the multiple original depth maps to establish the target optimization function.

2. The depth map processing method according to claim 1, characterized in that: The step of establishing a pixel matching relationship between every two original depth maps in the plurality of original depth maps according to the shooting parameters of the plurality of original depth maps specifically includes: Obtain the displacement parameters and device parameters of the shooting device; Determining pixel positions of the photographed subject in the plurality of original depth maps according to the displacement parameter and the device parameter; The pixel matching relationship is established according to the pixel positions.

3. The depth map processing method according to claim 1, characterized in that: The adjusting the pixel depth values ​​of the plurality of original depth maps specifically includes: Substituting the target model parameters into a preset correction model to obtain a target correction model; The pixel depth values ​​of the plurality of original depth maps are adjusted according to the target correction model.

4. The depth map processing method according to any one of claims 1 to 3, characterized in that: After adjusting the pixel depth values ​​of the plurality of original depth maps, the depth map processing method further includes: A plurality of point cloud data are generated according to the adjusted plurality of original depth maps, wherein the plurality of original depth maps correspond to the plurality of point cloud data in a one-to-one manner.

5. A depth map processing device, characterized in that: include: An acquisition unit, configured to acquire a plurality of original depth maps of a target scene at a plurality of shooting angles, wherein the plurality of shooting angles correspond to the plurality of original depth maps in a one-to-one manner; A processing unit, configured to establish a pixel matching relationship between every two original depth maps among the multiple original depth maps according to shooting parameters of the multiple original depth maps; The processing unit is further configured to adjust the pixel depth values ​​of the plurality of original depth maps according to a preset correction model and the pixel matching relationship; The processing unit is further used to optimize the model parameters of the preset correction model according to the multiple original depth maps and the pixel matching relationship; the processing unit is further used to divide the multiple pixel points in the multiple original depth maps into multiple groups according to the pixel matching relationship, wherein each group of pixel points includes two pixel points that match each other and are located in different original depth maps; substitute the depth value of each group of pixel points in the multiple groups of pixel points into the preset correction model to obtain multiple groups of target depth values; establish a target optimization function according to the multiple groups of target depth values; optimize the target optimization function according to a preset optimization algorithm to obtain optimized target model parameters; The preset correction model is expressed by the following formula: Where b is the baseline length of the depth camera, f is the focal length of the depth camera, p and q are the model parameters of the preset correction model, x is the horizontal coordinate of the image pixel in the original depth map, and d is the horizontal coordinate of the image pixel in the original depth map. raw is the original depth value of the image pixel corresponding to x, d new For d raw The corrected depth value obtained after the correction; The pixel matching relationships between the multiple original depth maps are traversed, and for each pair of pixel matching relationships, the preset correction model is respectively applied to the image pixels in the two original depth maps corresponding to the pixel matching relationship, and two corresponding corrected depth values ​​are obtained for each pair of matching pixels. For each pair of matching pixels, a sub-optimization function with model parameters as variables is established according to the difference between the corresponding two corrected depth values, and the sum of all sub-optimization functions is established according to each pair of matching pixels in the multiple original depth maps to establish the target optimization function.

6. An electronic device, characterized in that: include: The depth map processing apparatus as claimed in claim 5.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the depth map processing method according to any one of claims 1 to 4 are implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the depth map processing method according to any one of claims 1 to 4 are implemented.

Images