A method for acquiring a depth image, a terminal, and a computer-readable storage medium

Abstract

The embodiment of the present application discloses a depth image acquisition method and terminal, and a computer-readable storage medium, including: acquiring a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and acquiring a preset speckle reference map; using the preset speckle reference map and the speckle image, calculating first depth information and a first confidence corresponding to each pixel of the target object; the first confidence represents the accuracy of the first depth information; using the infrared grayscale image and the grayscale image obtained by graying the color image, calculating second depth information and a second confidence corresponding to each pixel of the target object; the second confidence represents the accuracy of the second depth information; according to the first confidence, the second confidence, the preset hole threshold, and the preset confidence threshold, the first depth information is fused with the second depth information to obtain the depth information of each pixel of the target object, and a depth image is obtained according to the depth information.

Description

Technical Field

[0001] The present application relates to photographing technology, and in particular to a depth image acquisition method and terminal, and a computer-readable storage medium. Background Art

[0002] A depth image refers to an image that uses the depth distance from the image acquisition device to each point of the target object as the pixel value, which can reflect the three-dimensional shape of the target object. At present, common methods for obtaining depth images include monocular structured light solutions, binocular shooting solutions, and binocular structured light solutions. The monocular structured light solution uses a projector to project a preset speckle with a special structure onto the target object. The infrared camera collects the speckle pattern on the target object, and coordinates are matched according to the speckle pattern to calculate the depth information and obtain the depth image. The binocular shooting solution uses the left and right cameras to shoot the same target object, and the depth information is calculated according to the parallax of the left and right cameras to obtain the depth image. The binocular structured light solution uses two infrared cameras to simultaneously collect the speckle pattern reflected by the target object to obtain the depth image.

[0003] However, when the monocular structured light solution is in strong light, or when the reflectivity of the target object is low, the infrared camera cannot collect the speckle pattern, and therefore cannot obtain depth information, resulting in poor depth image effect; when the binocular shooting solution is in low light scenes, or when the texture information of the target object is less, the images obtained by the left and right cameras contain less texture information, making it difficult to match pixels, resulting in poor depth image effect; and although the binocular structured light solution can improve the effect of the depth image in strong light and low reflectivity, it requires an additional infrared camera module, which increases the complexity of the software and hardware. Summary of the invention

[0004] The present application provides a depth image acquisition method and terminal, and a computer-readable storage medium, which can fuse depth information and improve the effect of the depth image without the need for an additional infrared camera module.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a depth image acquisition method, which is applied to a terminal, including:

[0007] Acquire a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and acquire a preset speckle reference image;

[0008] Calculating first depth information and a first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image; the first confidence level represents the accuracy of the first depth information;

[0009] Using the infrared grayscale image and the grayscale image obtained by graying the color image, calculate the second depth information and the second confidence corresponding to each pixel of the target object; the second confidence represents the accuracy of the second depth information;

[0010] According to the first confidence level, the second confidence level, a preset hole threshold and a preset confidence threshold, the first depth information is fused with the second depth information to obtain depth information of each pixel of the target object, and a depth image is obtained according to the depth information.

[0011] In the above solution, the terminal includes: a color camera, a speckle pattern transmitter, an infrared light transmitter and an infrared camera;

[0012] The color camera, the speckle pattern emitter, the infrared light emitter and the infrared camera are located on the same horizontal line or the same vertical line;

[0013] The distance between the color camera and the infrared camera is greater than the distance between the speckle pattern emitter and the infrared camera;

[0014] The speckle pattern emitter is adjacent to the color camera.

[0015] In the above solution, before acquiring the speckle image of the target object, the infrared grayscale image of the target object and the color image of the target object, and acquiring the preset speckle reference image, the method further includes:

[0016] Acquiring an initial speckle image of the target object, an initial infrared grayscale image of the target object, and an initial color image of the target object;

[0017] According to the resolution of the initial speckle image, the resolution of the initial color image is adjusted, and the coordinate system corresponding to the initial speckle image is aligned with the coordinate system corresponding to the initial color image to obtain an adjusted speckle image and an adjusted color image;

[0018] The adjusted speckle image, the initial infrared grayscale image and the adjusted color image are subjected to distortion correction processing, and the adjusted color image and the initial infrared grayscale image are subjected to epipolar correction to obtain the speckle image, the infrared grayscale image and the color image.

[0019] In the above solution, the step of acquiring the initial speckle image of the target object, the initial infrared grayscale image of the target object, and the initial color image of the target object includes:

[0020] collecting an initial speckle image projected by a speckle pattern emitter onto the target object through an infrared camera;

[0021] Using the infrared camera to collect an initial infrared grayscale image projected by the infrared light emitter onto the target object;

[0022] An initial color image of the target object is captured by a color camera.

[0023] In the above solution, the calculating the first depth information and the first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image includes:

[0024] Using a preset speckle matching algorithm, matching pixels corresponding to the i-th pixel of the target object in the speckle image from the preset speckle reference image to obtain a first matching pixel pair, and acquiring a first disparity value of the first matching pixel pair; wherein i is greater than or equal to 1;

[0025] Determine the first depth information corresponding to the i-th pixel of the target object by using a preset first depth conversion model, preset imaging parameters and the first disparity value;

[0026] Determining, according to the speckle image, the first confidence level corresponding to the i-th pixel of the target object;

[0027] Continue processing to obtain the first depth information and the first confidence corresponding to the i+1th pixel of the target object until i=N, and obtain the first depth information and the first confidence corresponding to each pixel of the target object; wherein N represents the total number of pixels of the target object in the speckle image, and N is a positive integer greater than 1.

[0028] In the above solution, determining the first confidence level corresponding to the i-th pixel of the target object according to the speckle image includes:

[0029] Determining, according to the speckle image, a local speckle energy density of an i-th pixel of the target object;

[0030] The first confidence level of the i-th pixel of the target object is determined by using a preset positive correlation between the local speckle energy density and the confidence level.

[0031] In the above solution, the use of the infrared grayscale image and the grayscale image obtained by graying the color image to calculate the second depth information and the second confidence corresponding to each pixel of the target object includes:

[0032] Using a preset grayscale matching algorithm, matching pixels corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image to obtain a second matching pixel pair, and acquiring a second disparity value of the second matching pixel pair; wherein i is greater than or equal to 1;

[0033] Determining the second depth information corresponding to the i-th pixel of the target object according to a preset second depth conversion model, preset imaging parameters and the second disparity value;

[0034] Determining the second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image;

[0035] Continue processing to obtain the second depth information and the second confidence corresponding to the i+1th pixel of the target object until i=N, and obtain the second depth information and the second confidence corresponding to each pixel; wherein N represents the total number of pixels of the target object in the infrared grayscale image, and N is a positive integer greater than 1.

[0036] In the above solution, determining the second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image includes:

[0037] Calculating a matching similarity cost of the second matching pixel pair according to the infrared grayscale image and the grayscale image;

[0038] The second confidence level corresponding to the i-th pixel of the target object is determined according to the matching similarity cost.

[0039] In the above solution, determining the second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image includes:

[0040] Calculating a first texture gradient of an i-th pixel of the target object in the grayscale image, and a second texture gradient of a pixel in the infrared grayscale image corresponding to an i-th pixel of the target object in the grayscale image;

[0041] The second confidence level of the i-th pixel of the target object is determined according to the first texture gradient and the second texture gradient.

[0042] In the above scheme, the first depth information is fused with the second depth information according to the first confidence level, the second confidence level, the preset hole threshold and the preset confidence threshold to obtain the depth information of each pixel of the target object, and the depth image is obtained according to the depth information, including:

[0043] Determine the primary depth information and secondary depth information of the i-th pixel of the target object according to the first confidence level, the second confidence level and the preset hole threshold; the primary depth information is the depth information with the largest number of pixels having a confidence level greater than the preset hole threshold; the secondary depth information is the remaining depth information of the first depth information and the second depth information except the primary depth information;

[0044] When the confidence level corresponding to the main depth information is not less than the preset confidence threshold, using the main depth information as the depth information of the i-th pixel of the target object;

[0045] When the confidence level corresponding to the primary depth information is less than the preset confidence threshold, calculating the depth information of the i-th pixel of the target object by using the primary depth information and the secondary depth information;

[0046] Continue to perform fusion processing on the i+1th pixel of the target object until i=N, obtain depth information of each pixel of the target object, and obtain a depth image according to the depth information.

[0047] In the above solution, determining the primary depth information and the secondary depth information of the i-th pixel of the target object according to the first confidence level, the second confidence level and the preset hole threshold value includes:

[0048] Counting the number of pixels whose first confidence level is greater than the preset hole threshold and the number of pixels whose second confidence level is greater than the preset hole threshold;

[0049] When the number of pixels whose first confidence level is greater than the preset hole threshold is not less than the number of pixels whose second confidence level is greater than the preset hole threshold, the first depth information of the i-th pixel of the target object is used as the primary depth information, and the second depth information of the i-th pixel of the target object is used as the secondary depth information;

[0050] When the number of pixels whose first confidence is greater than the preset hole threshold is less than the number of pixels whose second confidence is greater than the preset hole threshold, the second depth information of the i-th pixel of the target object is used as the main depth information, and the first depth information of the i-th pixel of the target object is used as the secondary depth information.

[0051] In the above solution, after acquiring the speckle image of the target object, the infrared grayscale image of the target object, the color image of the target object, and acquiring a preset speckle reference map, and before calculating the first depth information and the first confidence corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image, the method further includes:

[0052] Subtracting the infrared grayscale image from the speckle image to obtain an optimized speckle image;

[0053] Correspondingly, the method of calculating the first depth information and the first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image comprises:

[0054] The first depth information and the first confidence level corresponding to each pixel of the target object are calculated by using the preset speckle reference image and the optimized speckle image.

[0055] In a second aspect, an embodiment of the present application provides a terminal, including:

[0056] an acquisition unit, used to acquire a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and acquire a preset speckle reference image;

[0057] a processing unit, configured to calculate first depth information and a first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image; the first confidence level represents the accuracy of the first depth information; and calculate second depth information and a second confidence level corresponding to each pixel of the target object by using the infrared grayscale image and the grayscale image obtained by graying the color image; the second confidence level represents the accuracy of the second depth information;

[0058] A fusion unit is used to fuse the first depth information with the second depth information according to the first confidence level, the second confidence level, a preset hole threshold and a preset confidence threshold to obtain the depth information of each pixel of the target object, and obtain a depth image according to the depth information.

[0059] In a third aspect, an embodiment of the present application further provides a terminal, including:

[0060] The memory is used to store executable depth image acquisition instructions;

[0061] The processor is used to execute the executable depth image acquisition instructions stored in the memory to implement the method as described in any one of the first aspects above.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that it stores executable depth image acquisition instructions for causing a processor to execute and implement a method as described in any one of the first aspects above.

[0063] By adopting the above implementation manner, the terminal can calculate the first depth information of each pixel of the target object according to the speckle image of the target object and the preset speckle reference map, and then calculate the second depth information by using the infrared grayscale image of the target object and the grayscale image obtained by graying the color image of the target object, and then fuse the first depth information with the second depth information to obtain the final depth information. In addition, the infrared grayscale image and the speckle image of the target object can be collected by sharing an infrared camera. In this way, for each pixel of the target object, the terminal can make the first depth information and the second depth information complement each other, so as to improve the effect of the depth image without increasing the complexity of software and hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A process of a depth image acquisition method provided in an embodiment of the present application Figure 1 ;

[0065] Figure 2 A schematic diagram of a preset speckle reference image provided in an embodiment of the present application;

[0066] Figure 3 A process of a depth image acquisition method provided in an embodiment of the present application Figure 2 ;

[0067] FIG. 4( a ) is a timing diagram of a projection frequency and an exposure frequency provided in an embodiment of the present application;

[0068] FIG4( b ) is a schematic diagram of the arrangement of the speckle pattern emitter, the infrared light emitter, the infrared camera and the color camera provided in an embodiment of the present application;

[0069] FIG5( a ) is a schematic diagram of the epipolar lines of an infrared camera and a color camera before epipolar line correction provided in an embodiment of the present application;

[0070] FIG5( b ) is a schematic diagram of an initial infrared grayscale image before epipolar correction provided in an embodiment of the present application;

[0071] FIG5( c ) is a schematic diagram of an initial color image before epipolar correction provided in an embodiment of the present application;

[0072] FIG5( d ) is a schematic diagram of the epipolar lines of the infrared camera and the color camera after epipolar line correction provided in an embodiment of the present application;

[0073] FIG5( e ) is a schematic diagram of an infrared grayscale image after epipolar correction provided in an embodiment of the present application;

[0074] FIG5( f ) is a schematic diagram of a color image after epipolar correction provided in an embodiment of the present application;

[0075] Figure 6 A process of a depth image acquisition method provided in an embodiment of the present application Figure 3 ;

[0076] Figure 7 A schematic diagram of calculating first depth information provided by an embodiment of the present application;

[0077] Figure 8 Flow chart 4 of a depth image acquisition method provided in an embodiment of the present application;

[0078] Fig. 9 A schematic diagram of calculating second depth information provided by an embodiment of the present application;

[0079] Fig.10 Flow chart 5 of a depth image acquisition method provided in an embodiment of the present application;

[0080] Fig.11 A process of a depth image acquisition method provided in an embodiment of the present application Figure 6 ;

[0081] Fig.12 A schematic block diagram of a depth image acquisition method provided in an embodiment of the present application;

[0082] Fig.13 A schematic diagram of a flow chart of a depth image acquisition method provided in an embodiment of the present application;

[0083] Fig.14 A schematic diagram of a terminal structure provided in an embodiment of the present application Figure 1 ;

[0084] Fig.15 A schematic diagram of a terminal structure provided in an embodiment of the present application Figure 2 ;

[0085] Fig.16 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0087] A depth image refers to an image that uses the depth distance from the image acquisition device to each point of the target object as the pixel value. The depth image can reflect the three-dimensional shape of the target object. Common methods for acquiring depth images include monocular structured light solutions, binocular shooting solutions, and binocular structured light solutions. The monocular structured light solution uses a projector to project a preset speckle pattern with a special structure onto the target object. The infrared camera collects the speckle pattern projected onto the target object, and matches the pixels in the collected speckle pattern with the pixels in the preset speckle pattern to obtain matching coordinate pairs, and obtains the parallax of the matching coordinate pairs, and then uses the triangulation principle to calculate the depth information. Since the monocular structured light solution actively projects the speckle pattern, the monocular structured light belongs to the active depth image acquisition method. When using a monocular structured light solution to obtain depth information, it depends on the quality of the collected speckle pattern. When the light in the space is too strong or there are other infrared light sources in the space, the speckle pattern projected by the projector on the target object will be submerged, reducing the signal-to-noise ratio of the speckle pattern collected by the infrared camera. When the reflectivity of the target object is low, the speckle pattern projected by the projector on the target object cannot be reflected, making it impossible for the infrared camera to collect the speckle pattern. Both of these situations will result in the quality of the collected speckle pattern being low, making it difficult to match the coordinates of the collected speckle pattern with the preset speckle pattern, and the calculated depth information is inaccurate.

[0088] The binocular shooting scheme uses the left and right cameras to shoot the same target, matches the pixels of the images obtained by the left and right cameras, obtains matching coordinate pairs, and calculates the parallax of the matching coordinate pairs, thereby obtaining depth information based on the triangulation principle. Since the binocular shooting scheme passively receives the texture information of the target object, the binocular shooting scheme belongs to the passive depth image acquisition method. When the binocular shooting scheme performs coordinate matching, it matches according to the texture and other information in the images obtained by the left and right cameras. If the target object itself has less texture, or the target object is in a low-light scene, the texture information in the images obtained by the left and right cameras is less, and it is difficult to perform coordinate matching, making the calculated depth information inaccurate.

[0089] The binocular structured light solution uses a projector to project a speckle pattern with a special structure onto the target object, and then uses two infrared cameras to collect the speckle pattern on the target object. In strong light scenes, the speckle pattern is submerged. The binocular structure solution can degenerate the binocular shooting solution and perform coordinate matching based on the texture information of the images taken by the left and right infrared cameras to obtain depth information. In weak light scenes, the left and right speckle patterns are used to perform coordinate matching to obtain depth information. However, although the binocular structured light solution can improve the effect of depth images to a certain extent in strong light and low reflectivity, it requires an additional infrared camera module, which increases the cost of software and hardware.

[0090] Based on the problems existing in the above-mentioned method for obtaining a depth image, the basic idea of ​​the embodiments of the present application is to improve the effect of the depth image without increasing the complexity of software and hardware.

[0091] Embodiment 1

[0092] Based on the idea of ​​the above embodiment of the present application, the embodiment of the present application provides a depth image acquisition method, which can be applied to a terminal, see Figure 1 , the method may include:

[0093] S101, obtaining a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and obtaining a preset speckle reference image.

[0094] A depth image acquisition method provided in an embodiment of the present application is implemented in a scenario where a terminal is used to acquire a depth image of a target object. The terminal can acquire a speckle image of the target object, an infrared grayscale image of the target object, and a color image of the target object from its own storage space. At the same time, the terminal can acquire a preset speckle reference map from its own storage space, so that the speckle image, the infrared grayscale image, the color image, and the preset speckle reference map can be used to calculate depth information later.

[0095] In the embodiment of the present application, the terminal is provided with: a color camera, a speckle pattern transmitter, an infrared optical fiber transmitter and an infrared camera; wherein the color camera, the speckle pattern transmitter, the infrared light transmitter and the infrared camera are arranged on the same horizontal line or the same vertical line; the distance between the color camera and the infrared camera is greater than the distance between the speckle pattern transmitter and the infrared camera; the speckle pattern transmitter and the color camera are adjacent.

[0096] It should be noted that the speckle image of the target object is the image obtained after the initial speckle pattern collected by the terminal is processed by coordinate alignment, distortion correction and the like; the infrared grayscale image of the target object refers to the image obtained after the initial infrared grayscale image of the target object collected by the terminal is processed by epipolar correction; the color image of the target object refers to the image obtained after the initial color image of the target object collected by the terminal is processed by resolution adjustment, coordinate alignment, distortion correction and the like.

[0097] In the embodiment of the present application, the preset speckle reference image is an image obtained by projecting a designed speckle pattern onto an ideal plane at a certain distance from the speckle pattern emitter and then photographing it.

[0098] It is understandable that the preset speckle reference image in the embodiment of the present application is set in advance and stored in the terminal. In some embodiments of the present application, the speckle pattern in the preset speckle reference image can be set according to actual needs, and the embodiment of the present application is not limited here.

[0099] Exemplarily, an embodiment of the present application provides a schematic diagram of a preset speckle reference image, such as Figure 2 As shown in FIG. 1 , the preset speckle reference image is an image obtained by projecting the designed speckle pattern onto a plane 60 cm away from the speckle pattern emitter and then photographing it. The terminal stores the preset speckle reference image in its own storage space in advance for use when the terminal calculates the depth information of the target object.

[0100] In the embodiment of the present application, the terminal may be a terminal with computing functions such as a smart phone and a tablet computer, and the embodiment of the present application is not specifically limited here.

[0101] S102: Calculate first depth information and a first confidence level corresponding to each pixel of the target object by using a preset speckle reference map and a speckle image; the first confidence level represents the accuracy of the first depth information.

[0102] After acquiring the speckle image and the preset speckle reference map, the terminal can use the preset speckle reference map and the speckle image to calculate the first depth information corresponding to each pixel of the target object according to the triangulation principle. At the same time, the terminal can also calculate the accuracy of the first depth information corresponding to each pixel of the target object, that is, the first confidence of each pixel of the target object.

[0103] It should be noted that, since the target object is represented in pixels after being photographed by the terminal, the distance from each spatial point of the target object to the terminal can be regarded as the depth information that the pixel corresponding to the spatial point of the target object in the image taken by the terminal should have. Therefore, in the embodiment of the present application, the first depth information corresponding to each pixel of the target object refers to the depth information of each pixel of the target object in the speckle image, that is, the distance from the spatial point of the target object corresponding to each pixel of the target object to the terminal.

[0104] In the embodiment of the present application, when the terminal uses the preset speckle reference map and the speckle image to calculate the first depth information corresponding to each pixel of the target object, each pixel in the speckle image containing the target object is matched with the pixels in the preset speckle reference map one by one to obtain the matching pixel pairs corresponding to each pixel of the target object, and then the disparity values ​​of the matching pixel pairs are calculated, and then the first depth information of each pixel of the target object is calculated according to the triangulation ranging principle by using the preset first depth conversion model, the preset imaging parameters and the obtained disparity values, and finally the first confidence of each pixel is calculated according to the obtained speckle image.

[0105] It should be noted that, in the embodiment of the present application, the first confidence level is substantially the accuracy of matching each pixel of the target object in the speckle image with the pixel in the preset speckle reference map. Since the first depth information corresponding to each pixel of the target object is calculated by using the preset speckle reference map to match the speckle image with the pixel, and the disparity value of the matched pixel pair is obtained, when the pixel in the speckle image does not match the pixel in the preset speckle reference map accurately, the calculated depth information will also be inaccurate. Therefore, the accuracy of the first depth information can be characterized by the accuracy of the pixel matching between the speckle image and the preset speckle reference map, that is, the first confidence level.

[0106] S103, using the infrared grayscale image and the grayscale image obtained by graying the color image, calculate the second depth information and the second confidence corresponding to each pixel of the target object; the second confidence represents the accuracy of the second depth information.

[0107] After acquiring the infrared grayscale image and the color image of the target object, the terminal will first grayscale the color image to obtain a grayscale image, so that each pixel of the target object in the infrared grayscale image is matched with the pixel in the grayscale image to obtain the coordinate difference of the matching pixel pair, and then use the triangulation principle to calculate the second depth information of each pixel of the target object. At the same time, the accuracy of the second depth information corresponding to each pixel of the target object is calculated, that is, the second confidence of each pixel of the target object.

[0108] In the embodiment of the present application, the second depth information corresponding to each pixel of the target object refers to the depth information that each pixel of the target object in the infrared grayscale image should have, that is, the distance from the spatial point of the target object corresponding to each pixel point of the target object to the terminal.

[0109] It should be noted that since the feature information of pixels in the color image is quite different from that in the infrared grayscale image, in order to obtain the second matching pixel pair, the color image needs to be grayscaled to obtain a grayscale image, and then each pixel of the target object in the infrared grayscale image is pixel-matched with the obtained grayscale image to obtain the second matching pixel pair, and then the second depth information is calculated.

[0110] It can be understood that, similar to the first confidence level, the second confidence level is essentially the accuracy of matching each pixel of the target object in the infrared grayscale image with the pixel in the grayscale image. In the embodiment of the present application, the accuracy of the second depth information is characterized by the accuracy of the matching between the infrared grayscale image and the grayscale image pixels, i.e., the second confidence level.

[0111] S104: According to the first confidence level, the second confidence level, the preset hole threshold and the preset confidence threshold, the first depth information is merged with the second depth information to obtain depth information of each pixel of the target object, and a depth image is obtained according to the depth information.

[0112] After the terminal calculates the first depth information and the second depth information of each pixel of the target object, it fuses the first depth information and the second depth information corresponding to each pixel of the target object according to the first confidence level, the second confidence level, the preset hole threshold and the preset confidence threshold to obtain the depth information of each pixel of the target object. Then, the depth information is used as the pixel value, and after edge-preserving filtering, feathering and edge sharpening operations, the final depth image of the target object is obtained.

[0113] It should be noted that, when the terminal fuses the first depth information and the second depth information, it first determines the primary depth information and the secondary depth information for each pixel of the target object according to the existence of holes in the first depth information and the second depth information, and then judges whether the primary depth information of a certain pixel of the target object is accurate enough according to a preset confidence threshold. When the confidence of the primary depth information of a certain pixel of the target object is not lower than the preset confidence threshold, it indicates that the primary depth information of the pixel is very close to the real depth information that the pixel should have. At this time, there is no need to fuse the depth information. When the confidence of the primary depth information of a certain pixel of the target object is lower than the preset confidence threshold, it indicates that there is a deviation between the primary depth information of the pixel and the real depth information that the pixel should have. At this time, the secondary depth information of the pixel may be closer to the real depth information of the pixel. Therefore, the depth information of the pixel of the target object can be calculated based on the primary depth information and the secondary depth information of the pixel.

[0114] It should be noted that in the embodiment of the present application, a hole refers to a situation where the terminal believes that the calculated first depth information or second depth information is too inaccurate, that is, no reliable depth information is calculated, and the preset hole threshold is a parameter used to determine whether there is a hole in the first confidence and the second confidence of each pixel of the target object. Exemplarily, when the first confidence corresponding to a pixel of the target object is lower than the preset hole threshold, it indicates that the first depth information of the pixel is unreliable. At this time, the terminal can believe that no reliable first depth information of the pixel has been obtained, that is, the first depth information of the pixel is a hole.

[0115] In some embodiments of the present application, the preset hole threshold can be set according to actual needs, and the embodiments of the present application are not specifically limited here.

[0116] In the embodiment of the present application, the preset confidence threshold is a parameter used to determine whether the depth information is sufficiently accurate. Optionally, the preset confidence threshold can be set according to actual needs, and the embodiment of the present application does not specifically limit it here.

[0117] In the embodiment of the present application, the terminal can calculate the first depth information of each pixel of the target object according to the speckle image of the target object and the preset speckle reference map, and then calculate the second depth information by using the infrared grayscale image of the target object and the grayscale image obtained by graying the color image of the target object, and then fuse the first depth information with the second depth information to obtain the final depth information, and the infrared grayscale image and the speckle image of the target object can share an infrared camera for acquisition, so that for each pixel of the target object, the terminal can make the first depth information and the second depth information complement each other, so as to improve the effect of the depth image without increasing the software and hardware costs.

[0118] Embodiment 2

[0119] Based on the same concept as the first embodiment, Figure 1 , see Figure 3 Before obtaining the speckle image of the target object, the infrared grayscale image of the target object and the color image of the target object in the first embodiment, and obtaining the preset speckle reference image, that is, before S101 in the first embodiment, the method may further include: S105-S107, as follows:

[0120] S105 , collecting an initial speckle image of the target object, an initial infrared grayscale image of the target object, and an initial color image of the target object.

[0121] Before acquiring the speckle image, infrared grayscale image and color image of the target object, the terminal needs to first acquire the initial speckle image, initial infrared grayscale image and initial color image of the target object, and process the acquired images to obtain the speckle image, infrared grayscale image and color image that can be directly used for calculation.

[0122] In the embodiment of the present application, the terminal can collect an initial color image of the target object through a color camera arranged on the terminal, collect an initial infrared grayscale image of the target object through an infrared light emitter and an infrared camera, and collect an initial speckle image of the target object through a speckle pattern emitter and an infrared camera.

[0123] In some embodiments of the present application, the specific implementation process of acquiring the initial speckle image of the target object, the initial infrared grayscale image of the target object, and the initial color image of the target object, that is, S105, may include: S1051-S1053, as follows:

[0124] S1051. Collect an initial speckle image projected by the speckle pattern emitter onto the target object through an infrared camera.

[0125] The terminal can project a pre-designed speckle pattern onto a target object through a speckle pattern transmitter arranged on the terminal, and then the terminal captures the speckle pattern reflected by the target object through an infrared camera arranged on the terminal, and uses the captured image as an initial speckle image.

[0126] Since the infrared camera needs to collect the initial infrared grayscale image in addition to the initial speckle image, in the embodiment of the present application, the terminal can set the exposure frequency of the infrared camera to twice the projection frequency of the speckle pattern emitter, and set the infrared light emitter and the speckle pattern emitter to alternately project infrared light and speckle pattern to the target object at the same projection frequency. In this way, the terminal alternately collects the initial speckle image and the initial infrared grayscale image through the infrared camera.

[0127] It should be noted that when the exposure frequency of the infrared camera is high, the initial infrared grayscale image and the initial speckle image can be regarded as being acquired approximately synchronously.

[0128] In some embodiments of the present application, the exposure frequency of the infrared camera can be set according to actual needs, and the embodiments of the present application are not limited here.

[0129] S1052: Using an infrared camera to capture an initial infrared grayscale image projected by the infrared light emitter onto the target object.

[0130] After the terminal collects the initial speckle image through the infrared camera in the previous exposure, it will then use the infrared light transmitter to project infrared light to the target object, and use the infrared camera to collect the infrared light reflected by the target object, and use the collected image as the initial infrared grayscale image.

[0131] It should be noted that the projection frequency of the infrared light emitter is the same as the projection frequency of the speckle pattern emitter, which is half of the exposure frequency of the infrared camera.

[0132] S1053: Capture an initial color image of the target object through a color camera.

[0133] While the terminal uses the infrared camera to collect the initial infrared grayscale image, the terminal will also use the color camera to collect the initial color image of the target object, so that the initial infrared grayscale image and the initial color image are collected at the same time to ensure that the infrared grayscale image obtained after processing is the same as the target object in the color image, so that the infrared grayscale image and the grayscale image obtained after graying the color image can better match pixels.

[0134] It should be noted that, since the initial infrared grayscale image and the initial color image are collected simultaneously, the exposure frequency of the color camera needs to be the same as the projection frequency of the infrared light emitter, which is half of the exposure frequency of the infrared camera.

[0135] Illustratively, an embodiment of the present application provides a timing diagram of the projection frequencies of a speckle pattern emitter and an infrared light emitter, and the exposure frequencies of an infrared camera and a color camera. As shown in FIG4(a), the speckle pattern emitter, the infrared light emitter, the infrared camera, and the color camera are horizontally arranged on the terminal according to the arrangement shown in FIG4(b). As can be seen from FIG4(a), the exposure frequency of the infrared camera is twice the projection frequency of the speckle pattern emitter, the projection frequency of the infrared light emitter and the exposure frequency of the color camera. The speckle pattern emitter and the infrared light emitter project to the target object alternately at the same projection frequency. The exposure frequency of the color camera is the same as the projection frequency of the infrared light emitter, and the color camera and the infrared light emitter always operate simultaneously. In this way, the terminal can use the speckle pattern emitter, the infrared light emitter, the infrared camera and the color camera to collect the initial infrared grayscale image, the initial speckle image and the initial color image of the target object according to the working sequence in FIG4(a), so as to facilitate the subsequent processing of the initial infrared grayscale image, the initial speckle image and the initial color image to obtain the speckle image, the infrared grayscale image and the color image.

[0136] S106, adjusting the resolution of the initial color image according to the resolution of the initial speckle image, and aligning the coordinate system corresponding to the initial speckle image with the coordinate system corresponding to the initial color image to obtain an adjusted speckle image and an adjusted color image.

[0137] Since the initial speckle image and the initial color image are collected by different cameras, the resolution and coordinate system of the initial speckle image and the initial color image are very likely to be different. When the terminal fuses the first depth information and the second depth information, the terminal must ensure that the first depth information and the second depth information of the same pixel are fused. Therefore, after collecting the initial speckle image, the initial infrared grayscale image and the initial color image, the terminal needs to adjust the initial color image and the initial speckle image to the same resolution, and align the coordinate systems of the initial speckle image and the initial color image to obtain the adjusted speckle image and the adjusted color image.

[0138] It should be noted that, in general, the resolution of a color camera is greater than that of an infrared camera. Therefore, in order to ensure that the initial speckle image with a lower resolution is not distorted, the terminal may scale the resolution of the initial color image to the resolution of the initial speckle image.

[0139] Exemplarily, when the resolution of the initial speckle image is 640×480, the terminal may scale the resolution of the initial color image to 640×480.

[0140] In the embodiment of the present application, before the terminal aligns the coordinate system corresponding to the initial speckle image with the coordinate system of the initial color image, it is also necessary to perform dual-camera calibration on the infrared camera and the color camera.

[0141] In some embodiments of the present application, the terminal may perform dual-camera calibration on the infrared camera and the color camera using the Zhang Zhengyou calibration method, or may perform dual-camera calibration on the infrared camera and the color camera using other methods that can achieve the same purpose.

[0142] S107, performing distortion correction processing on the adjusted speckle image, the initial infrared grayscale image and the adjusted color grayscale image, and performing epipolar correction on the adjusted color image and the initial infrared grayscale image to obtain a speckle image, an infrared grayscale image and a color image.

[0143] After the terminal obtains the adjusted speckle image and the adjusted color image, the terminal also needs to perform distortion correction processing on the adjusted speckle image and the adjusted infrared image, as well as the initial infrared grayscale image that has not been processed. After that, the terminal also needs to perform epipolar correction on the adjusted color image and the initial infrared grayscale image to obtain the speckle image, the infrared grayscale image and the color image, so that there is no distortion caused by the camera lens process in the speckle image, the infrared grayscale image and the color image, and the same pixel point of the target object is highly consistent in the infrared grayscale image and the color image, so as to facilitate pixel matching of the infrared grayscale image and the grayscale image obtained after the color image is grayed.

[0144] It can be understood that before the terminal performs epipolar correction on the adjusted color image and the initial infrared grayscale image, the epipolar line of the infrared camera set on the terminal is not parallel to the epipolar line of the color camera, so that the position of the same pixel point of the target object in the adjusted color image is different from the position of the pixel point in the initial infrared grayscale image. After the terminal performs epipolar correction on the adjusted color image and the initial infrared grayscale image, the height of the same pixel point of the target object in the infrared grayscale image is the same as the height in the color image, which is equivalent to the terminal collecting the infrared grayscale image and the color image when the epipolar line of the infrared camera and the epipolar line of the color camera are parallel to each other. In this way, when performing pixel matching between the infrared grayscale image and the grayscale image obtained after the grayscale color image is grayed, it is only necessary to find matching pixels on the same row.

[0145] Exemplarily, an embodiment of the present application provides a schematic diagram of epipolar correction, referring to FIG5(a), the infrared camera 2 and the color camera 3 are both arranged on the terminal plane 1, FIG5(b) is the initial infrared grayscale image acquired by the terminal through the infrared camera 2, and FIG5(c) is the initial color image acquired by the terminal through the color camera 3. Before the terminal performs epipolar correction, the epipolar lines of the infrared camera 2 and the color camera 3 are not parallel, and thus the position of the pixel 1 in FIG5(b) is different from the position of the pixel 1 in FIG5(c). Figure 5(d) is a schematic diagram of the infrared camera 2 and the color camera 3 after epipolar correction. At this time, the epipolar lines of the infrared camera 2 and the color camera 3 are parallel to each other. Correspondingly, Figure 5(e) is equivalent to the image obtained by the infrared camera 2 after epipolar correction, and Figure 5(f) is equivalent to the image obtained by the color camera 3 after epipolar correction. The height of pixel 1 in Figure 5(e) is the same as the height of pixel 1 in Figure 5(f). Therefore, when matching pixels, it is only necessary to find corresponding pixels in the same row, which reduces the amount of calculation for pixel matching.

[0146] In the embodiment of the present application, the terminal can collect an initial speckle image and an initial infrared grayscale image through an infrared camera, and collect an initial color image through a color camera, and perform a series of processing such as resolution adjustment, coordinate alignment, dedistortion, and epipolar correction on the obtained initial speckle image, initial infrared grayscale image, and initial color image to obtain a speckle image, an infrared grayscale image, and a color image, ensuring that the terminal can calculate the first depth information and the second depth information of each pixel of the target object using the speckle image, the infrared grayscale image, and the color image, and fuse the first depth information with the second depth information.

[0147] In some embodiments of the present application, after acquiring a speckle image of the target object, an infrared grayscale image of the target object, and a color image of the target object, and acquiring a preset speckle reference map, and before calculating the first depth information and the first confidence corresponding to each pixel of the target object using the preset speckle reference map and the speckle image, that is, after S101 and before S102 in the first embodiment, the method further includes: S108, as follows:

[0148] S108, subtracting the infrared grayscale image from the speckle image to obtain an optimized speckle image.

[0149] Since the initial speckle image obtained by the terminal through the infrared camera may contain not only the speckle pattern and infrared light reflected by the target object and the light emitted by the initial infrared grayscale image transmitter, but also the infrared components in the ambient light reflected by the target object, these infrared components will affect the pixel matching between the speckle image and the preset speckle reference image. Therefore, after acquiring the speckle image, infrared grayscale image and color image of the target object and before calculating the first depth corresponding to each pixel of the target object, the terminal can also subtract the infrared grayscale image from the speckle image, and use the obtained image as the optimized speckle image.

[0150] At this time, correspondingly, using the preset speckle reference map and the speckle image to calculate the first depth information and the first confidence level corresponding to each pixel of the target object includes: using the preset speckle reference map and the optimized speckle image to calculate the first depth information and the first confidence level corresponding to each pixel of the target object.

[0151] It is understandable that, since the terminal will also collect the infrared component of the ambient light reflected by the target object when collecting the initial infrared grayscale image through the infrared camera, the terminal can subtract the infrared grayscale image from the speckle image, so that the infrared component of the ambient light in the speckle image can be eliminated, thereby improving the signal-to-noise ratio of the speckle image.

[0152] In the embodiment of the present application, the terminal can subtract the infrared grayscale image from the speckle image, eliminate the infrared component in the ambient light reflected by the target object, and improve the signal-to-noise ratio of the speckle image, so that when the terminal uses the speckle image and the preset speckle reference map for pixel matching, the adaptability to scenes with strong ambient light such as outdoor scenes is improved.

[0153] Embodiment 3

[0154] Based on the same concept of embodiment 1, see Figure 6 In the first embodiment, the preset speckle reference map and the speckle image are used to calculate the first depth information and the first confidence level corresponding to each pixel of the target object, that is, the specific implementation process of S102 in the first embodiment may include: S1021-S1024, as follows:

[0155] S1021. Using a preset speckle matching algorithm, match pixels corresponding to the i-th pixel of the target object in the speckle image from a preset speckle reference image to obtain a first matching pixel pair, and acquire a first disparity value of the first matching pixel pair; wherein i is greater than or equal to 1.

[0156] After acquiring the speckle image and the preset speckle reference image, the terminal needs to use the preset speckle matching algorithm to match the pixel corresponding to the i-th pixel of the target object in the speckle image from the preset speckle reference image to obtain a first matching pixel pair, and subtract the coordinates of two pixels in the first matching pixel pair to obtain a first disparity value corresponding to the first matching pixel pair, so that the terminal can subsequently calculate the first depth information corresponding to the i-th pixel of the target object according to the first disparity value.

[0157] It should be noted that the preset speckle matching algorithm in the embodiment of the present application may use the shape of the speckle pattern around the i-th pixel of the target object in the speckle image to search for a speckle pattern with the same shape in the preset speckle reference image, thereby determining the pixel corresponding to the i-th pixel of the target object from the preset speckle reference image. Alternatively, the pixel corresponding to the i-th pixel of the target object may be determined from the preset speckle reference image by other methods that can achieve the same purpose.

[0158] Optionally, the specific implementation of the preset speckle matching algorithm can be designed according to actual needs, and the embodiment of the present application is not limited here.

[0159] It should be noted that, in the embodiment of the present application, the first disparity value refers to the number of pixels that differ between two pixels in the first matching pixel pair. Exemplarily, when the terminal uses a preset matching algorithm to determine that the fifth pixel in the preset speckle reference image matches the first pixel of the target object in the speckle image, the terminal uses the first pixel of the target object in the speckle image and the fifth pixel in the preset speckle reference image as the first matching pixel pair. At this time, the first disparity value of the first matching pixel pair is 4.

[0160] S1022: Determine first depth information corresponding to an i-th pixel of the target object by using a preset first depth conversion model, preset imaging parameters, and a first disparity value.

[0161] After obtaining the first disparity value, the terminal may calculate a first depth value corresponding to the i-th pixel of the target object according to the triangulation principle by using a pre-stored preset first depth conversion model, preset imaging parameters and the first disparity value.

[0162] In the embodiment of the present application, the preset first depth conversion model refers to a triangulation formula for calculating depth information using the first disparity value and the preset imaging parameters. After obtaining the preset imaging parameters and the first disparity value, the terminal can substitute the preset imaging parameters and the first disparity value into the preset first depth conversion model to calculate the first depth information. It should be noted that the preset imaging parameters in the embodiment of the present application may include the baseline distance, focal length, standard plane distance, and the side length of each pixel.

[0163] Illustratively, the present application embodiment provides a schematic diagram of calculating the first depth information, such as Figure 7 As shown in the figure, C is the position of the infrared camera, D is the position of the speckle pattern emitter, CD is the baseline distance, F is the position of the target object, Dis is the distance between the infrared camera C and the speckle pattern emitter D and the target object, that is, the first depth information corresponding to the i-th pixel of the target object, the distance between the infrared camera C and the speckle pattern emitter D and the reference plane is L, A is the imaging point of the speckle pattern projected by the speckle pattern emitter and reflected by the target object, that is, the pixel position of the speckle pattern in the speckle image, B is the imaging point of the speckle pattern projected by the speckle pattern emitter and reflected by the reference plane when there is no target object, that is, the pixel position of the speckle pattern in the preset speckle reference map, and the plane where A and B are located is the imaging plane. At this time, the terminal can calculate the first disparity value n according to the pixel position of the speckle pattern in the speckle image and the pixel position of the speckle pattern in the preset speckle reference map, and the distance between AB is the product of the disparity n and the side length p of each pixel, that is, np. At this time, after the terminal knows the baseline distance CD, the focal length f, the standard plane distance L, the first disparity value n and the side length p of each pixel, the first depth conversion model is preset as shown in formula (1):

[0164]

[0165] The terminal can calculate the first depth information Dis according to formula (1). It should be noted that since the first disparity value n is obtained by subtracting the coordinates of two pixels in the first matching pixel pair, the sign of the first disparity value n can be negative. Figure 7 As shown in FIG. 1 , when the target object F is located on the left side of the reference plane, that is, the distance between the infrared camera C and the speckle pattern emitter D and the target object is less than the distance between the infrared camera C and the speckle pattern emitter D and the reference plane, the sign of the first disparity value n is positive; if the target object is located on the right side of the reference plane, that is, the distance between the infrared camera C and the speckle pattern emitter D and the target object is greater than the distance between the infrared camera C and the speckle pattern emitter D and the reference plane, the distance of the first disparity value is negative.

[0166] S1023: Determine a first confidence level corresponding to the i-th pixel of the target object according to the speckle image.

[0167] After calculating the first depth information of the i-th pixel of the target object, the terminal needs to determine the first confidence level corresponding to the i-th pixel of the target object according to the speckle image, so as to subsequently fuse the first depth information with the second depth information according to the first confidence level and the second confidence level.

[0168] In some embodiments of the present application, in the embodiments of the present application, the first confidence corresponding to the i-th pixel of the target object is determined according to the speckle image, which can be determined according to S1023a-S1023b as follows:

[0169] S1023a: Determine the local speckle energy density of the i-th pixel of the target object according to the speckle image.

[0170] Since the first confidence level corresponding to the i-th pixel of the target object is essentially the accuracy of matching the i-th pixel of the target object in the speckle image with the pixel in the preset speckle reference image, and when the local speckle energy density of the i-th pixel of the target object is high, it indicates that there are more speckle patterns around the i-th pixel of the target object in the speckle image, or the speckle pattern is clearer. At this time, the richer the features of the i-th pixel of the target object in the speckle image, the more accurately the pixel corresponding to the i-th pixel of the target object can be matched from the preset speckle reference image. Therefore, the terminal can determine the local speckle energy density of the i-th pixel of the target object according to the speckle image, and then determine the accuracy of matching the i-th pixel of the target object in the speckle image with the pixel in the preset speckle reference image.

[0171] In some embodiments of the present application, the specific calculation method of the local speckle energy density can be set by the research and development personnel according to actual needs, and the embodiments of the present application are not specifically limited here.

[0172] S1023b. Determine a first confidence level of the i-th pixel of the target object by using a preset positive correlation between the local speckle energy density and the confidence level.

[0173] After calculating the local speckle energy density corresponding to the i-th pixel of the target object, the terminal may determine the first confidence level of the i-th pixel of the target object by using the preset positive correlation between the local speckle energy density and the confidence level.

[0174] In some embodiments of the present application, the specific correlation degree of the preset positive correlation between the local speckle energy density and the confidence level can be set according to actual needs.

[0175] At this point, the terminal has completed the specific process of determining the first confidence level corresponding to the i-th pixel of the target object according to the speckle image.

[0176] S1024. Continue processing to obtain the first depth information and the first confidence level corresponding to the i+1th pixel of the target object until i=N, and obtain the first depth information and the first confidence level corresponding to each pixel of the target object; wherein N represents the total number of pixels of the target object in the speckle image, and N is a positive integer greater than 1.

[0177] After the terminal calculates the first depth information and the first confidence corresponding to the i-th pixel of the target object, the terminal will then process the i+1-th pixel of the target object and obtain the first depth information and the first confidence corresponding to the i+1-th pixel of the target object, until i=N, completing the processing of all pixels of the target object and obtaining the first depth information and the first confidence corresponding to each pixel of the target object.

[0178] In the embodiment of the present application, the terminal can perform pixel matching on the speckle image and the preset speckle reference image according to a preset speckle matching algorithm to obtain first depth information and a first confidence level of each pixel of the target object, so that the subsequent terminal can fuse the first depth information with the second depth information according to the first confidence level and the second confidence level.

[0179] In some embodiments of the present application, see Figure 8 In the first embodiment, the infrared grayscale image and the grayscale image obtained by graying the color image are used to calculate the second depth information and the second confidence corresponding to each pixel of the target object, that is, the specific implementation process of S103 in the first embodiment may include: S1031-S1034, as follows:

[0180] S1031. Use a preset grayscale matching algorithm to match pixels corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image to obtain a second matching pixel pair, and acquire a second disparity value of the second matching pixel pair; wherein i is greater than or equal to 1.

[0181] After acquiring the infrared grayscale image and the color image, the terminal will first grayscale the color image to obtain a grayscale image, and then use a preset grayscale matching algorithm to match the pixel corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image to obtain a second matching pixel pair, and subtract the coordinates of the two pixels in the second matching pixel pair to obtain a second disparity value, so that the terminal can subsequently calculate the second depth information based on the second disparity value.

[0182] It should be noted that in the embodiment of the present application, the terminal uses a preset grayscale matching algorithm to perform pixel matching based on the texture information in the infrared grayscale image and the grayscale image. Optionally, the specific implementation of the preset grayscale matching algorithm can be set according to actual needs, and the embodiment of the present application is not specifically limited here.

[0183] S1032: Determine second depth information corresponding to an i-th pixel of the target object according to a preset second depth conversion model, preset imaging parameters, and a second disparity value.

[0184] After calculating the second disparity value of the second matching pixel pair, the terminal can use the preset second depth conversion model, preset imaging parameters and the second disparity value to determine the second depth information corresponding to the i-th pixel of the target object according to the triangulation principle.

[0185] In the embodiment of the present application, the preset second depth conversion model refers to a triangulation formula for calculating depth information using the second disparity value and the preset imaging parameters. After obtaining the second disparity value and the preset imaging parameters, the terminal can substitute the second disparity value and the preset imaging parameters into the preset second depth conversion model to calculate the second depth information.

[0186] It should be noted that the preset imaging parameters in the embodiments of the present application may include: baseline distance and focal length.

[0187] Exemplarily, the present application embodiment provides a schematic diagram of calculating the second depth information, such as Fig. 9 As shown, O R is the location of the infrared camera, O T is the location of the color camera, O R With O T The distance between them is b, that is, the baseline distance b, P is the location of the target object, P1 is the image point obtained when the terminal photographs the target object P through the infrared camera, P'1 is the image point obtained when the terminal photographs the target object P through the color camera, x R is the coordinate of the target object’s image point P1 in the infrared grayscale image, x T is the coordinate of the image point P1' of the target object in the grayscale image, f is the focal length of the infrared camera and the color camera, and Z is the distance from the infrared camera and the color camera to the target object, that is, the depth to be calculated. At this time, equation (2) can be obtained from similar triangles:

[0188]

[0189] Furthermore, the preset second depth model can be derived from formula (2), as shown in formula (3):

[0190]

[0191] Wherein, d is the second disparity value. It can be seen from formula (3) that as long as the terminal knows the baseline distance b, the focal length f and the second disparity value d, the second depth information Z can be calculated according to the preset second depth conversion model.

[0192] S1033. Determine a second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image.

[0193] After calculating the second depth information corresponding to the i-th pixel of the target object, the terminal also needs to determine the second confidence level corresponding to the i-th pixel of the target object based on the infrared grayscale image and the grayscale image, so as to facilitate the subsequent fusion of the first depth information and the second depth information based on the first confidence level and the second confidence level.

[0194] In some embodiments of the present application, the second confidence level corresponding to the i-th pixel of the target object is determined according to the infrared grayscale image and the grayscale image in the embodiment of the present application, which can be determined according to S1033a-S1033b as follows:

[0195] S1033a. Calculate the matching similarity cost of the second matching pixel pair based on the infrared grayscale image and the grayscale image.

[0196] When the terminal matches the pixel corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image according to the preset grayscale matching algorithm, the i-th pixel of the target object in the grayscale image is calculated with each pixel in the infrared grayscale image for the matching similarity cost, and the pixel point with the smallest calculated matching similarity cost is identified as the pixel corresponding to the i-th pixel of the target object in the grayscale image. Therefore, the terminal can determine the second confidence of the i-th pixel of the target object by matching the similarity cost.

[0197] In some embodiments of the present application, the specific calculation method of the matching similarity cost can be set according to actual needs, and the embodiments of the present application are not specifically limited here.

[0198] S1033b. Determine a second confidence level corresponding to the i-th pixel of the target object according to the matching similarity cost.

[0199] After calculating the matching similarity cost of the second matching pixel pair, the terminal can determine the second confidence level corresponding to the i-th pixel of the target object according to the matching similarity cost.

[0200] In some embodiments of the present application, the terminal can set a cost threshold and compare the matching similarity cost of the second matching pixel pair with the cost threshold. When the matching similarity cost is greater than the cost threshold, the terminal believes that even if the pixel with the smallest matching similarity cost in the infrared grayscale image is used as the second matching pixel pair of the i-th pixel of the grayscale image target object, there is still a high probability that these two pixels will be mismatched. At this time, the confidence of the second depth information is low.

[0201] In some embodiments of the present application, the terminal may also compare the matching similarity cost of the second matching pixel pair with other matching similarity costs. When the matching similarity cost of the second matching pixel pair is much smaller than other matching similarity costs, the terminal considers that the confidence of the second depth information is higher; when the matching similarity cost of the second matching pixel pair is not much different from other matching similarity costs, the terminal considers that the confidence of the second depth information is lower.

[0202] In some embodiments of the present application, the second confidence level corresponding to the i-th pixel of the target object is determined according to the infrared grayscale image and the grayscale image in the embodiment of the present application, and can also be determined according to S1033c-S1033d, as follows:

[0203] S1033c, calculating a first texture gradient of an i-th pixel of the target object in the grayscale image, and a second texture gradient of a pixel in the infrared grayscale image corresponding to the i-th pixel of the target object in the grayscale image.

[0204] When the terminal determines the second confidence level corresponding to the i-th pixel of the target object based on the infrared grayscale image and the grayscale image, it can also determine the second confidence level of the i-th pixel of the target object based on the calculated texture information of the i-th pixel of the target object in the grayscale image and the calculated texture information of the pixel corresponding to the i-th pixel of the target object in the infrared grayscale image.

[0205] In some embodiments of the present application, the specific method for calculating the texture gradient can be selected according to actual needs, and the embodiments of the present application are not limited here.

[0206] S1033d. Determine a second confidence level of the i-th pixel of the target object according to the first texture gradient and the second texture gradient.

[0207] After calculating the first texture gradient and the second texture gradient, the terminal can compare the first texture gradient with the second texture gradient. When the first texture gradient is equal to the second texture gradient, or the difference between the first texture gradient and the second texture gradient is very small, the terminal believes that the second confidence corresponding to the i-th pixel of the target object is higher. When the difference between the first texture gradient and the second texture gradient is large, the terminal believes that the second confidence corresponding to the i-th pixel of the target object is lower.

[0208] At this point, the terminal has completed the process of determining the second confidence level corresponding to the i-th pixel of the target object based on the infrared grayscale image and the grayscale image.

[0209] S1034. Continue processing to obtain the second depth information and the second confidence level corresponding to the i+1th pixel of the target object until i=N, and obtain the second depth information and the second confidence level corresponding to each pixel; wherein N represents the total number of pixels of the target object in the infrared grayscale image, and N is a positive integer greater than 1.

[0210] After calculating the second depth information and the second confidence corresponding to the i-th pixel of the target object, the terminal will immediately process the i+1-th pixel of the target object to obtain the second depth information and the second confidence corresponding to the i+1-th pixel of the target object, until i=N, completing the processing of all pixels of the target object to obtain the second depth information and the second confidence corresponding to each pixel of the target object.

[0211] In an embodiment of the present application, the terminal can perform pixel matching on the grayscale image obtained by graying the infrared grayscale image and the color image according to a preset grayscale matching algorithm, and obtain a second disparity value for each pixel of the target object, and then calculate the second depth information and second confidence for each pixel of the target object. In this way, the terminal can fuse the first depth information with the second depth information according to the first confidence and the second confidence.

[0212] In some embodiments of the present application, see Fig.10 In one embodiment, the first depth information is fused with the second depth information according to the first confidence level, the second confidence level, the preset hole threshold and the preset confidence threshold to obtain the depth information of each pixel of the target object, and a depth image is obtained according to the depth information. That is, the implementation process of S104 in the first embodiment may include: S1041-S1044, as follows:

[0213] S1041. Determine the primary depth information and secondary depth information of the i-th pixel of the target object according to the first confidence level, the second confidence level and the preset hole threshold; the primary depth information is the depth information with the largest number of pixels whose confidence level is greater than the preset hole threshold; the secondary depth information is the depth information remaining in the first depth information and the second depth information except the primary depth information.

[0214] After calculating the first confidence level and the second confidence level, the terminal will first determine the presence of holes in all pixels of the target object using the first confidence level, the second confidence level and a preset hole threshold, and determine depth information with a better overall effect for all pixels of the target object based on the number of pixels with holes in the first depth information among all pixels of the target object and the number of pixels with holes in the second depth information among all pixels of the target object. The terminal uses the depth information to determine primary depth information for each pixel of the target object, and uses the remaining depth information to determine secondary depth information for each pixel of the target object, i.e., the primary depth information and secondary depth information are determined for the i-th pixel of the target object.

[0215] In some embodiments of the present application, Fig.11 As shown, in the embodiment of the present application, the primary depth information and the secondary depth information of the i-th pixel of the target object are determined according to the first confidence level, the second confidence level and the preset hole threshold, which can be implemented according to S1041a-S1041c as follows:

[0216] S1041a. Count the number of pixels whose first confidence level is greater than a preset hole threshold and the number of pixels whose second confidence level is greater than the preset hole threshold.

[0217] When the terminal determines the depth information with better overall effect for all pixels of the target object based on the number of holes, it is necessary to first count the number of holes in the first depth information of all pixels of the target object and the number of spaces in the second depth information of all pixels of the target object. Since the terminal compares the confidence with the preset hole threshold to determine whether the depth information corresponding to the pixel is a hole, the terminal only needs to count the number of pixels whose first confidence is greater than the preset hole threshold to obtain the number of pixels whose first depth information is not a hole among all pixels of the target object. Similarly, the terminal only needs to count the number of pixels whose second confidence is greater than the preset hole threshold to obtain the number of pixels whose second depth information is not a hole among all pixels of the target object, so as to facilitate the subsequent determination of the primary depth information and the secondary depth information for each pixel of the target object.

[0218] S1041b. When the number of pixels whose first confidence level is greater than a preset hole threshold is not less than the number of pixels whose second confidence level is greater than the preset hole threshold, the first depth information of the i-th pixel of the target object is used as the primary depth information, and the second depth information of the i-th pixel of the target object is used as the secondary depth information.

[0219] When the number of pixels whose first confidence is greater than the preset hole threshold is not less than the number of pixels whose second confidence is greater than the preset hole threshold, it indicates that for all pixels of the target object, the overall effect of the first depth information is better. Therefore, the first depth information of the i-th pixel of the target object is used as the main depth information, and the second depth information of the i-th pixel of the target object is used as the secondary depth information, so that for each pixel of the target object, the first depth information is the main depth information, and the second depth information is the secondary depth information, so that the final depth information of the i-th pixel of the target object can be determined according to the main depth information, the secondary depth information and the preset confidence threshold.

[0220] S1041c. When the number of pixels whose first confidence is greater than the preset hole threshold is less than the number of pixels whose second confidence is greater than the preset hole threshold, the second depth information of the i-th pixel of the target object is used as the primary depth information, and the first depth information of the i-th pixel of the target object is used as the secondary depth information.

[0221] When the number of pixels whose first confidence is greater than the preset hole threshold is less than the number of pixels whose second confidence is greater than the preset hole threshold, it means that the overall effect of the second depth information is better for all pixels of the target object. At this time, the second depth information of the i-th pixel of the target object is used as the main depth information, and the first depth information of the i-th pixel of the target object is used as the secondary depth information, so that for each pixel of the target object, the second depth information is the main depth information, and the first depth information is the secondary depth information.

[0222] At this point, the terminal has completed the process of determining the primary depth information and the secondary depth information of the i-th pixel of the target object according to the first confidence level, the second confidence level and the preset hole threshold.

[0223] S1042: When the confidence level corresponding to the main depth information is not less than a preset confidence threshold, the main depth information is used as the depth information of the i-th pixel of the target object.

[0224] After the terminal determines the primary depth information and secondary depth information of the i-th pixel of the target object, it is also necessary to compare the confidence of the primary depth information with the preset confidence threshold to determine whether the primary depth information of the i-th pixel of the target object is sufficiently accurate, that is, sufficiently close to the real depth information corresponding to the i-th pixel of the target object. When the terminal determines that the confidence corresponding to the primary depth information of the i-th pixel of the target object is greater than or equal to the preset confidence threshold, it indicates that the primary depth information of the pixel is sufficiently accurate. At this time, the primary depth information of the i-th pixel of the target object is directly used as the final depth information of the i-th pixel of the target object.

[0225] S1043: When the confidence level corresponding to the primary depth information is less than a preset confidence threshold, the primary depth information and the secondary depth information are used to determine the depth information of the i-th pixel of the target object.

[0226] When the terminal determines that the confidence corresponding to the primary depth information of the i-th pixel of the target object is less than a preset confidence threshold, it indicates that the primary depth information of the pixel may not be accurate enough. Therefore, it is necessary to use the primary depth information and the secondary depth information to jointly calculate the depth information of the i-th pixel of the target object.

[0227] In some embodiments of the present application, the terminal weights the primary depth information and the secondary depth information to calculate the depth information of the i-th pixel of the target object, or the terminal averages the primary depth information and the secondary depth information to calculate the depth information of the i-th pixel of the target object.

[0228] S1044 , continue fusing the i+1th pixel of the target object until i=N, obtain depth information of each pixel of the target object, and obtain a depth image according to the depth information.

[0229] After obtaining the depth information of the i-th pixel of the target object, the terminal will continue to fuse the depth information of the i+1-th pixel of the target object until i=N, completing the fusion of the depth information of each pixel of the target object and obtaining the depth information of each pixel of the target object. The obtained depth information of each pixel of the target object is used as the pixel value, and after a series of filtering processes, the final depth image is obtained.

[0230] Exemplarily, an embodiment of the present application provides a schematic block diagram of a method for obtaining a depth image, such as Fig.12 As shown, the terminal collects the initial speckle image projected on the target object by the speckle pattern emitter 3 through the infrared camera 2, collects the initial infrared grayscale image projected on the target object by the infrared light emitter 4 through the infrared camera 2, and collects the initial color image of the target object through the color camera 1. The initial speckle image, the initial infrared grayscale image and the initial color image are processed by resolution adjustment, coordinate system alignment, distortion correction and epipolar correction to obtain the speckle image, infrared grayscale image and color image. After that, the terminal uses a preset speckle matching algorithm to perform pixel matching on the speckle image and the speckle reference image, calculates the first depth information and the first confidence of each pixel of the target object, uses a preset grayscale matching algorithm to perform pixel matching on the infrared grayscale image and the grayscale image, and then calculates the second depth information and the second confidence of each pixel of the target object. Finally, according to the first confidence and the second confidence, a depth fusion operation is performed to obtain the final depth information of each pixel of the target object.

[0231] Exemplarily, the present application embodiment also provides a flowchart of a method for obtaining a depth image, such as Fig.13 As shown, the terminal obtains a speckle image 01, an infrared grayscale image 02 and a color image 03 of the target object, and performs background subtraction 05 on the speckle image to obtain an optimized speckle image. Then, a preset speckle matching algorithm 06 is used to match the pixels corresponding to the i-th pixel of the target object in the optimized speckle image in the preset speckle reference image 04, and a disparity value is calculated, and then the first depth information and the first confidence 07 of the i-th pixel of the target object are calculated. At the same time, the terminal performs preprocessing 08 on the color image 03 to obtain a grayscale image, and then uses a preset grayscale matching algorithm 09 to match the pixels corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image, and a disparity value is calculated, and then the second depth information and the second confidence 10 of the i-th pixel of the target object are calculated. Finally, the obtained first depth information and the second depth information are depth-fused 11 to obtain the final depth information corresponding to the i-th pixel of the target object. When the terminal obtains the final depth information corresponding to all pixels of the target object, a depth image 12 is obtained and output according to the depth information.

[0232] Embodiment 4

[0233] Based on the same concept as in Embodiments 1 to 3, Fig.14 As shown, an embodiment of the present application provides a terminal 1, which may include:

[0234] An acquisition unit 10 is used to acquire a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and to acquire a preset speckle reference image;

[0235] The processing unit 11 is configured to calculate first depth information and a first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image; the first confidence level represents the accuracy of the first depth information; and calculate second depth information and a second confidence level corresponding to each pixel of the target object by using the infrared grayscale image and the grayscale image obtained by graying the color image; the second confidence level represents the accuracy of the second depth information;

[0236] The fusion unit 12 is used to fuse the first depth information with the second depth information according to the first confidence level, the second confidence level, a preset hole threshold and a preset confidence threshold to obtain the depth information of each pixel of the target object, and obtain a depth image according to the depth information.

[0237] In some embodiments of the present application, the terminal 1 includes: a color camera, a speckle pattern transmitter, an infrared light transmitter and an infrared camera;

[0238] The color camera, the speckle pattern emitter, the infrared light emitter and the infrared camera are located on the same horizontal line or the same vertical line;

[0239] The distance between the color camera and the infrared camera is greater than the distance between the speckle pattern emitter and the infrared camera;

[0240] The speckle pattern emitter is adjacent to the color camera.

[0241] In some embodiments of the present application, Fig.15 As shown, the terminal 1 further includes: a collection unit 13;

[0242] The acquisition unit 13 is used to acquire an initial speckle image of the target object, an initial infrared grayscale image of the target object, and an initial color image of the target object;

[0243] The processing unit 11 is further configured to adjust the resolution of the initial color image according to the resolution of the initial speckle image, and align a coordinate system corresponding to the initial speckle image with a coordinate system corresponding to the initial color image to obtain an adjusted speckle image and an adjusted color image; and perform distortion correction processing on the adjusted speckle image, the initial infrared grayscale image, and the adjusted color image, and perform epipolar correction on the adjusted color image and the initial infrared grayscale image to obtain the speckle image, the infrared grayscale image, and the color image.

[0244] In some embodiments of the present application, the acquisition unit 13 is specifically used to acquire an initial speckle image projected on the target object by the speckle pattern emitter through an infrared camera; acquire an initial infrared grayscale image projected on the target object by the infrared light emitter through the infrared camera; and acquire an initial color image of the target object through a color camera.

[0245] In some embodiments of the present application, the processing unit 11 is specifically configured to match pixels corresponding to the i-th pixel of the target object in the speckle image from the preset speckle reference image using a preset speckle matching algorithm to obtain a first matching pixel pair, and acquire a first disparity value of the first matching pixel pair; wherein i is greater than or equal to 1; determine the first depth information corresponding to the i-th pixel of the target object using a preset first depth conversion model, preset imaging parameters, and the first disparity value; determine the first confidence corresponding to the i-th pixel of the target object according to the speckle image; and continue to process to obtain the first depth information and the first confidence corresponding to the i+1-th pixel of the target object until i=N, to obtain the first depth information and the first confidence corresponding to each pixel of the target object; wherein N represents the total number of pixels of the target object in the speckle image, and N is a positive integer greater than 1.

[0246] In some embodiments of the present application, the processing unit 11 is specifically configured to determine, according to the speckle image, a local speckle energy density of an i-th pixel of the target object; and determine the first confidence level of the i-th pixel of the target object by using a preset positive correlation between the local speckle energy density and the confidence level.

[0247] In some embodiments of the present application, the processing unit 11 is specifically used to use a preset grayscale matching algorithm to match the pixel corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image, obtain a second matching pixel pair, and obtain a second disparity value of the second matching pixel pair; wherein i is greater than or equal to 1; determine the second depth information corresponding to the i-th pixel of the target object according to a preset second depth conversion model, preset imaging parameters and the second disparity value; determine the second confidence corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image; and continue processing to obtain the second depth information and the second confidence corresponding to the i+1-th pixel of the target object until i=N, to obtain the second depth information and the second confidence corresponding to each pixel; wherein N represents the total number of pixels of the target object in the infrared grayscale image, and N is a positive integer greater than 1.

[0248] In some embodiments of the present application, the processing unit 11 is specifically used to calculate the matching similarity cost of the second matching pixel pair based on the infrared grayscale image and the grayscale image; and determine the second confidence corresponding to the i-th pixel of the target object based on the matching similarity cost.

[0249] In some embodiments of the present application, the processing unit 11 is specifically used to calculate the first texture gradient of the i-th pixel of the target object in the grayscale image, and the second texture gradient of the pixel in the infrared grayscale image corresponding to the i-th pixel of the target object in the grayscale image; determine the second confidence of the i-th pixel of the target object based on the first texture gradient and the second texture gradient.

[0250] In some embodiments of the present application, the fusion unit 12 is specifically used to determine the main depth information and secondary depth information of the i-th pixel of the target object according to the first confidence, the second confidence and the preset hole threshold; the main depth information is the depth information with the largest number of pixels whose confidence is greater than the preset hole threshold; the secondary depth information is the depth information remaining in the first depth information and the second depth information except the main depth information; when the confidence corresponding to the main depth information is not less than the preset confidence threshold, the main depth information is used as the depth information of the i-th pixel of the target object; when the confidence corresponding to the main depth information is less than the preset confidence threshold, the main depth information and the secondary depth information are used to calculate the depth information of the i-th pixel of the target object; and continue to perform fusion processing on the i+1-th pixel of the target object until i=N, to obtain the depth information of each pixel of the target object, and obtain a depth image based on the depth information.

[0251] In some embodiments of the present application, the fusion unit 12 is specifically used to count the number of pixels whose first confidence is greater than the preset hole threshold, and the number of pixels whose second confidence is greater than the preset hole threshold; when the number of pixels whose first confidence is greater than the preset hole threshold is not less than the number of pixels whose second confidence is greater than the preset hole threshold, the first depth information of the i-th pixel of the target object is used as the main depth information, and the second depth information of the i-th pixel of the target object is used as the secondary depth information; when the number of pixels whose first confidence is greater than the preset hole threshold is less than the number of pixels whose second confidence is greater than the preset hole threshold, the second depth information of the i-th pixel of the target object is used as the main depth information, and the first depth information of the i-th pixel of the target object is used as the secondary depth information.

[0252] In some embodiments of the present application, the processing unit 11 is further configured to subtract the infrared grayscale image from the speckle image to obtain an optimized speckle image; accordingly, the first depth information and the first confidence corresponding to each pixel of the target object are calculated by using the preset speckle reference image and the speckle image; comprising:

[0253] The first depth information and the first confidence level corresponding to each pixel of the target object are calculated by using the preset speckle reference image and the optimized speckle image.

[0254] Embodiment 5

[0255] Based on the same concept of the first embodiment and the third embodiment, Fig.16 A schematic diagram of the structure of a terminal proposed in an embodiment of the present application is shown in FIG. Fig.16 As shown, a terminal proposed in the present application may include a processor 01 and a memory 02 storing executable instructions of the processor 01. The processor 01 is used to execute the executable depth image acquisition instructions stored in the memory to implement the method in any one or more embodiments of Embodiments 1 to 3.

[0256] In some embodiments of the present application, the terminal includes: a color camera 03, a speckle pattern transmitter 04, an infrared light transmitter 05, and an infrared camera 06;

[0257] The color camera 03, the speckle pattern emitter 04, the infrared light emitter 05 and the infrared camera 06 are located on the same horizontal line or the same vertical line;

[0258] The distance between the color camera 03 and the infrared camera 05 is greater than the distance between the speckle pattern emitter 04 and the infrared camera 05;

[0259] The speckle pattern emitter 04 is adjacent to the color camera 03 .

[0260] In an embodiment of the present application, the processor 01 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the function of the processor may also be other, and the embodiment of the present application is not specifically limited. The terminal also includes a memory 02, which may be connected to the processor 01, wherein the memory 02 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0261] In practical applications, the memory 02 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 01.

[0262] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0263] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program codes.

[0264] An embodiment of the present application provides a computer-readable storage medium on which executable depth image acquisition instructions are stored, which is applied to a terminal. When the program is executed by a processor, the method in any one or more of embodiments 1 to 3 is implemented.

[0265] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0266] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0267] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0268] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0269] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0270] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A depth image acquisition method, applied to a terminal, characterized in that: The method comprises: Acquire a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and acquire a preset speckle reference image; Calculating first depth information and a first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image; the first confidence level represents the accuracy of the first depth information; Using the infrared grayscale image and the grayscale image obtained by graying the color image, calculate the second depth information and the second confidence corresponding to each pixel of the target object; the second confidence represents the accuracy of the second depth information; According to the first confidence level, the second confidence level, a preset hole threshold and a preset confidence threshold, the first depth information is merged with the second depth information to obtain depth information of each pixel of the target object, and a depth image is obtained according to the depth information; The step of fusing the first depth information with the second depth information according to the first confidence level, the second confidence level, a preset hole threshold, and a preset confidence threshold to obtain depth information of each pixel of the target object, and obtaining a depth image according to the depth information includes: Determine the primary depth information and secondary depth information of the i-th pixel of the target object according to the first confidence level, the second confidence level and the preset hole threshold; the primary depth information is the depth information with the largest number of pixels having a confidence level greater than the preset hole threshold; the secondary depth information is the remaining depth information of the first depth information and the second depth information except the primary depth information; When the confidence level corresponding to the main depth information is not less than the preset confidence threshold, using the main depth information as the depth information of the i-th pixel of the target object; When the confidence level corresponding to the primary depth information is less than the preset confidence threshold, calculating the depth information of the i-th pixel of the target object by using the primary depth information and the secondary depth information; Continue to perform fusion processing on the i+1th pixel of the target object until i=N, obtain depth information of each pixel of the target object, and obtain a depth image according to the depth information.

2. The method according to claim 1, characterized in that The terminal comprises: a color camera, a speckle pattern transmitter, an infrared light transmitter and an infrared camera; The color camera, the speckle pattern emitter, the infrared light emitter and the infrared camera are located on the same horizontal line or the same vertical line; The distance between the color camera and the infrared camera is greater than the distance between the speckle pattern emitter and the infrared camera; The speckle pattern emitter is adjacent to the color camera.

3. The method according to claim 1, characterized in that: Before acquiring the speckle image of the target object, the infrared grayscale image of the target object and the color image of the target object, and acquiring a preset speckle reference image, the method further includes: Acquiring an initial speckle image of the target object, an initial infrared grayscale image of the target object, and an initial color image of the target object; According to the resolution of the initial speckle image, the resolution of the initial color image is adjusted, and the coordinate system corresponding to the initial speckle image is aligned with the coordinate system corresponding to the initial color image to obtain an adjusted speckle image and an adjusted color image; The adjusted speckle image, the initial infrared grayscale image and the adjusted color image are subjected to distortion correction processing, and the adjusted color image and the initial infrared grayscale image are subjected to epipolar correction to obtain the speckle image, the infrared grayscale image and the color image.

4. The method according to claim 3, characterized in that The acquiring of the initial speckle image of the target object, the initial infrared grayscale image of the target object and the initial color image of the target object comprises: collecting an initial speckle image projected by a speckle pattern emitter onto the target object through an infrared camera; Using the infrared camera to collect an initial infrared grayscale image projected by the infrared light emitter onto the target object; An initial color image of the target object is captured by a color camera.

5. The method according to claim 1, characterized in that The calculating the first depth information and the first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image includes: Using a preset speckle matching algorithm, matching pixels corresponding to the i-th pixel of the target object in the speckle image from the preset speckle reference image to obtain a first matching pixel pair, and acquiring a first disparity value of the first matching pixel pair; wherein i is greater than or equal to 1; Determine the first depth information corresponding to the i-th pixel of the target object by using a preset first depth conversion model, preset imaging parameters and the first disparity value; Determining, according to the speckle image, the first confidence level corresponding to the i-th pixel of the target object; Continue processing to obtain the first depth information and the first confidence corresponding to the i+1th pixel of the target object until i=N, and obtain the first depth information and the first confidence corresponding to each pixel of the target object; wherein N represents the total number of pixels of the target object in the speckle image, and N is a positive integer greater than 1.

6. The method according to claim 5, characterized in that Determining the first confidence level corresponding to the i-th pixel of the target object according to the speckle image includes: Determining, according to the speckle image, a local speckle energy density of an i-th pixel of the target object; The first confidence level of the i-th pixel of the target object is determined by using a preset positive correlation between the local speckle energy density and the confidence level.

7. The method according to claim 1, characterized in that The step of calculating the second depth information and the second confidence level corresponding to each pixel of the target object by using the infrared grayscale image and the grayscale image obtained by graying the color image includes: Using a preset grayscale matching algorithm, matching pixels corresponding to the i-th pixel of the target object in the grayscale image from the infrared grayscale image to obtain a second matching pixel pair, and acquiring a second disparity value of the second matching pixel pair; wherein i is greater than or equal to 1; Determining the second depth information corresponding to the i-th pixel of the target object according to a preset second depth conversion model, preset imaging parameters and the second disparity value; Determining the second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image; Continue processing to obtain the second depth information and the second confidence corresponding to the i+1th pixel of the target object until i=N, and obtain the second depth information and the second confidence corresponding to each pixel; wherein N represents the total number of pixels of the target object in the infrared grayscale image, and N is a positive integer greater than 1.

8. The method according to claim 7, characterized in that Determining the second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image includes: Calculating a matching similarity cost of the second matching pixel pair according to the infrared grayscale image and the grayscale image; The second confidence level corresponding to the i-th pixel of the target object is determined according to the matching similarity cost.

9. The method according to claim 7, characterized in that: Determining the second confidence level corresponding to the i-th pixel of the target object according to the infrared grayscale image and the grayscale image includes: Calculating a first texture gradient of an i-th pixel of the target object in the grayscale image, and a second texture gradient of a pixel in the infrared grayscale image corresponding to an i-th pixel of the target object in the grayscale image; The second confidence level of the i-th pixel of the target object is determined according to the first texture gradient and the second texture gradient.

10. The method according to claim 1, characterized in that The determining, according to the first confidence level, the second confidence level and the preset hole threshold, the primary depth information and the secondary depth information of the i-th pixel of the target object comprises: Counting the number of pixels whose first confidence level is greater than the preset hole threshold and the number of pixels whose second confidence level is greater than the preset hole threshold; When the number of pixels whose first confidence level is greater than the preset hole threshold is not less than the number of pixels whose second confidence level is greater than the preset hole threshold, the first depth information of the i-th pixel of the target object is used as the primary depth information, and the second depth information of the i-th pixel of the target object is used as the secondary depth information; When the number of pixels whose first confidence is greater than the preset hole threshold is less than the number of pixels whose second confidence is greater than the preset hole threshold, the second depth information of the i-th pixel of the target object is used as the main depth information, and the first depth information of the i-th pixel of the target object is used as the secondary depth information.

11. The method according to claim 1, characterized in that: After acquiring the speckle image of the target object, the infrared grayscale image of the target object, the color image of the target object, and acquiring a preset speckle reference map, and before calculating the first depth information and the first confidence corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image, the method further includes: Subtracting the infrared grayscale image from the speckle image to obtain an optimized speckle image; Correspondingly, the method of calculating the first depth information and the first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image comprises: The first depth information and the first confidence level corresponding to each pixel of the target object are calculated by using the preset speckle reference image and the optimized speckle image.

12. A terminal, characterized in that: The terminal comprises: an acquisition unit, used to acquire a speckle image of a target object, an infrared grayscale image of the target object, and a color image of the target object, and acquire a preset speckle reference image; a processing unit, configured to calculate first depth information and a first confidence level corresponding to each pixel of the target object by using the preset speckle reference map and the speckle image; the first confidence level represents the accuracy of the first depth information; and calculate second depth information and a second confidence level corresponding to each pixel of the target object by using the infrared grayscale image and the grayscale image obtained by graying the color image; the second confidence level represents the accuracy of the second depth information; A fusion unit is used to determine the main depth information and secondary depth information of the i-th pixel of the target object according to the first confidence, the second confidence and a preset hole threshold; the main depth information is the depth information with the largest number of pixels whose confidence is greater than the preset hole threshold; the secondary depth information is the depth information remaining in the first depth information and the second depth information except the main depth information; when the confidence corresponding to the main depth information is not less than the preset confidence threshold, the main depth information is used as the depth information of the i-th pixel of the target object; when the confidence corresponding to the main depth information is less than the preset confidence threshold, the main depth information and the secondary depth information are used to calculate the depth information of the i-th pixel of the target object; and the fusion processing is continued on the i+1-th pixel of the target object until i=N, so as to obtain the depth information of each pixel of the target object, and obtain a depth image according to the depth information.

13. A terminal, characterized in that: The terminal includes: a memory and a processor; The memory is used to store executable depth image acquisition instructions; The processor is used to execute the executable depth image acquisition instructions stored in the memory to implement the method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: An executable depth image acquisition instruction is stored, which is used to cause a processor to execute and implement the method described in any one of claims 1-11.

Images