A method, apparatus and electronic device for obtaining depth images

Abstract

An embodiment of the present invention provides a method, apparatus, and electronic device for obtaining a depth image. Wherein, the method includes: for each camera, obtaining a rectified original image captured by the camera in multiple exposures with different structured light, where there are at least two cameras, and different cameras are located at different positions; according to the same encoding rule, for each camera, encoding the original image captured by the camera to obtain at least one encoded image corresponding to the camera; matching the encoded images corresponding to all the cameras to obtain a target depth image. It is possible to improve the accuracy of the determined depth image without reducing the matching window.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, apparatus, and electronic device for obtaining a depth image. Background Art

[0002] In the application scenarios of three-dimensional detection, it is necessary to obtain a depth image recording depth information. Since there is a parallax in the images captured by different cameras at the same spatial position, and the parallax is related to the depth of the spatial position, the depth information can be calculated based on the parallax to generate a depth image.

[0003] To determine the parallax, it is necessary to determine the same spatial position in the images captured by different cameras. In the related art, structured light with a specific spatial distribution is used to irradiate the scene, and multiple images are captured by cameras located at different positions. The parallax is determined by matching these multiple images.

[0004] The larger the matching window used for matching, the lower the accuracy of the determined parallax, resulting in a lower accuracy of the depth image. Therefore, in the related art, to improve the accuracy of the depth image, a matching window with a smaller size is often selected. However, the smaller the matching window, the lower the matching success rate. Therefore, if the matching window is too small, the parallax cannot be determined. It can be seen that reducing the size of the matching window can only improve the accuracy of the depth image to a certain extent.

[0005] Therefore, further improving the accuracy of the depth image has become an urgent technical problem to be solved. Summary of the Invention

[0006] The purpose of the embodiments of the present invention is to provide a method, apparatus, and electronic device for obtaining a depth image to improve the accuracy of the obtained depth image. The specific technical solutions are as follows:

[0007] In the first aspect of the embodiments of the present invention, a method for obtaining a depth image is provided. The method includes:

[0008] For each camera, obtain the rectified original images captured by the camera in multiple exposures with different structured light, where there are at least two cameras, and different cameras are located at different positions;

[0009] According to the same encoding rule, for each of the cameras, encode the original image captured by the camera to obtain at least one encoded image corresponding to the camera, where the size of the encoded image is the same as that of the original image, each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image, and the original pixel rows corresponding to at least two encoded pixel rows are in different original images, and each encoded pixel row is used to record the pixel values of the corresponding original pixel row;

[0010] Match the encoded images corresponding to all the cameras to obtain a target depth image.

[0011] In a possible embodiment, the original pixel rows corresponding to every two adjacent pixel rows in the encoded image are in different original images.

[0012] In a possible embodiment, the matching the encoded images corresponding to all the cameras to obtain a target depth image includes:

[0013] Match the encoded images in at least one encoded image group to obtain a first image corresponding to the encoded image group, where each encoded image group includes a plurality of the encoded images, and different encoded images in each encoded image group correspond to different cameras and are obtained according to the same encoding method, and the first image is a depth image or a disparity image;

[0014] Based on one of the first images, obtain a target depth image; or, fuse all the first images to obtain a target depth image.

[0015] In a possible embodiment, the fusing all the first images to obtain a depth image includes:

[0016] If the first image is a depth image, fuse all the first images to obtain a first fusion result as the target depth image;

[0017] If the first image is a disparity image, fuse all the first images to obtain a second fusion result; perform depth calculation according to the second fusion result to obtain a target depth image.

[0018] In a possible embodiment, the encoding the original image captured by the camera to obtain at least one encoded image corresponding to the camera includes:

[0019] Repeat the following steps until an encoded image is obtained:

[0020] For each row coordinate, select one original pixel row at the row coordinate from all the original images captured by the camera;

[0021] Record the image information of the selected original pixel row in the encoded pixel row located at the row coordinate.

[0022] In a possible embodiment, selecting an original pixel row located at the row coordinate from all the original images captured by the camera includes:

[0023] Select the original pixel row with the most valid information and located at the row coordinate from all the original images captured by the camera.

[0024] In a possible embodiment, for each camera, obtaining the rectified original images captured by the camera in multiple exposures with different structured lights includes:

[0025] For each camera, obtain the images captured by the camera in multiple exposures with different structured lights as the images to be processed;

[0026] For each camera, preprocess the images to be processed captured by the camera to obtain the original images, where the preprocessing includes one or more of upsampling, pixel merging, smoothing processing, and denoising processing.

[0027] In the second aspect of the embodiments of the present invention, a depth image acquisition device is provided, and the device includes:

[0028] An original image acquisition module, configured to obtain, for each camera, the rectified original images captured by the camera in multiple exposures with different structured lights, where there are at least two cameras, and different cameras are located at different positions;

[0029] An encoding module, configured to encode the original images captured by the camera according to the same encoding rule to obtain at least one encoded image corresponding to the camera, where the size of the encoded image is the same as that of the original image, each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image, and the original pixel rows corresponding to at least two adjacent encoded pixel rows are in different original images, and each encoded pixel row is used to record the pixel value of the corresponding original pixel row;

[0030] A matching module, configured to match the encoded images corresponding to all the cameras to obtain a target depth image.

[0031] In the third aspect of the embodiments of the present invention, an electronic device is provided, and the electronic device includes a plurality of cameras, a plurality of structured light sources, a processor, and a memory;

[0032] The multiple structured light sources are used to sequentially project structured light with different spatial distributions for multiple exposures;

[0033] The multiple cameras are used to respectively capture original images during the multiple exposures;

[0034] The memory is used to store a computer program;

[0035] When the processor is used to execute the program stored in the memory, the method steps described in any one of the above first aspects are implemented.

[0036] In the fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the above first aspects are implemented.

[0037] Advantages of the embodiments of the present invention:

[0038] The depth image acquisition method, device and electronic device provided by the embodiments of the present invention encode multiple images captured by the same camera during multiple exposures, so as to combine multiple original images into an encoded image in units of pixel rows. Since adjacent pixel rows in the encoded image record the original pixel rows in different original images, when matching the encoded image, the image information of the original pixel rows from multiple original images is integrated simultaneously in the matching window. Therefore, the density of the image information is higher, and the number of pixel points required to be included in the matching window during matching is smaller. Therefore, the size of the matching window can be further reduced, and the obtained depth image is more accurate.

[0039] Of course, when implementing any product or method of the present invention, it is not necessarily required to achieve all the above advantages simultaneously. Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0041] Figure 1 It is a schematic flowchart of a depth image acquisition method provided by an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of image encoding provided by an embodiment of the present invention;

[0043] Figure 3 It is a schematic flowchart of an image matching method provided by an embodiment of the present invention;

[0044] Figure 4 A schematic flowchart of the image encoding method provided by the embodiment of the present invention;

[0045] Figure 5 A schematic structural diagram of the depth image acquisition device provided by the embodiment of the present invention;

[0046] Figure 6 A schematic structural diagram of the electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on this application belong to the scope of protection of the present invention.

[0048] To more clearly illustrate the depth image acquisition method provided by the embodiment of the present invention, the following will give an exemplary illustration of a possible application scenario of the depth image acquisition method provided by the embodiment of the present invention. It can be understood that the following examples are only a possible application scenario of the depth image acquisition method provided by the embodiment of the present invention. In other possible application scenarios, the depth image acquisition method provided by the embodiment of the present invention can also be applied to other possible application scenarios, and the following examples do not impose any restrictions on this.

[0049] The relative positions of the same spatial position with respect to cameras located at different spatial positions are different. Therefore, the same spatial position is located at different coordinates in the images captured by cameras located at different spatial positions. In this article, the coordinate difference of the same spatial position in the images captured by different cameras is called parallax. Based on the principle of optical imaging, the parallax of the same spatial position is related to the depth of this spatial position. On the basis of knowing the imaging parameters of the camera, the depth of the spatial position can be calculated according to the parallax.

[0050] Therefore, in the related art, parallax is often used to calculate the depth, so as to obtain a depth image. To determine the parallax, it is necessary to determine the coordinates of the same spatial position in the images captured by different cameras. To distinguish different spatial positions, a device with a projection structured light is used to project structured light onto a specific area. Since the light intensity of the structured light has a certain distribution in space, the gray levels presented by different spatial positions in the captured images are different. If the gray levels of multiple pixel regions in different images match, it can be considered that these multiple pixel regions are obtained by different cameras photographing the same spatial position, so as to determine the coordinates of the same spatial position in the images captured by different cameras.

[0051] Exemplarily, assume that there are two cameras, denoted as the left camera and the right camera respectively. If the gray levels of the first pixel region in the left camera image captured by the left camera match those of the second pixel region in the right camera image captured by the right camera, it is considered that the first pixel region and the second pixel region are obtained by photographing the same spatial position, and this spatial position is denoted as spatial position A. Then the coordinates of spatial position A in the first image are located within the first pixel region, while the coordinates of spatial position A in the second image are located within the second pixel region.

[0052] It can be seen that through image matching, the coordinates of spatial position A in the left camera image and the right camera image cannot be accurately determined, but only the ranges of the coordinates of spatial position A in the left camera image and the right camera image can be determined. Moreover, the smaller the first pixel region and the second pixel region are, the smaller the ranges of the coordinates of spatial position A in the left camera image and the right camera image are, and thus the more accurate the determined coordinates are. Based on the more accurate coordinates, a more accurate parallax can be calculated, and then a more accurate depth information can be obtained. It can be seen that to make the determined depth information more accurate, it is necessary to make the first pixel region and the second pixel region smaller, and the sizes of the first pixel region and the second pixel region depend on the matching window used for matching. Therefore, the matching window needs to be as small as possible.

[0053] However, the smaller the matching window is, the smaller the first pixel region and the second pixel region are, resulting in less image information contained in the first pixel region and the second pixel region. Therefore, it is easy to cause false matching, that is, determining that the first pixel region and the second pixel region with actually unmatched gray levels are gray level matched. Therefore, to accurately perform image matching, the matching window needs to be larger than a specific threshold to ensure that the first pixel region and the second pixel region include enough image information.

[0054] Since the matching window needs to be larger than a specific threshold, the matching window can only be reduced to a certain extent, resulting in that by reducing the matching window, the accuracy of the determined depth information can only be improved to a certain extent. When the matching window is reduced to the specific threshold, it will no longer be possible to further improve the accuracy of the determined depth information by reducing the matching window.

[0055] Based on this, an embodiment of the present invention provides a method for obtaining a depth image, as Figure 1 shown, including:

[0056] S101, for each camera, obtain the rectified original images captured by the camera in multiple exposures with different structured lights.

[0057] S102, according to the same coding rule, for each camera, encode at least one original image captured by the camera to obtain the encoded image corresponding to the camera.

[0058] Among them, the size of the encoded image is the same as that of the original image. Each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image. At least two encoded pixel rows correspond to the original pixel rows in different original images, and each encoded pixel row is used to record the pixel values of the corresponding original pixel row.

[0059] S103. Match the encoded images corresponding to all the cameras to obtain a target depth image.

[0060] Selecting this embodiment, by encoding multiple images captured by the same camera in multiple exposures, multiple original images are combined into an encoded image in units of pixel rows. Since at least two pixel rows in the encoded image record the original pixel rows in different original images, when matching the encoded image, the image information of the original pixel rows from multiple original images is integrated in the matching window at the same time. Therefore, the density of the image information is higher, and the number of pixel points required to be included in the matching window during matching is less. Therefore, the size of the matching window can be further reduced, and the obtained depth image is more accurate.

[0061] The steps of S101 - S103 described above will be described below:

[0062] In S101, the number of cameras may be different according to different application scenarios, but there should be at least two cameras. Any two cameras in this article can be integrated on the same device or independent of each other. Exemplarily, two cameras may refer to two cameras integrated on a binocular camera.

[0063] The cameras being located at different positions means that the positions of the optical centers of the camera lenses are different. Based on parallax principle, the optical axes of each camera lens should be as parallel as possible, and each lens should be set on a straight line perpendicular to the optical axis. For the convenience of description in the following text, only the case of two cameras is described. For the case of three or more cameras, the principle is the same and will not be elaborated here.

[0064] The resolution of each original image should be the same, and the imaging parameters such as camera position, azimuth angle, and focal length should remain unchanged during shooting of multiple original images. The original image was an image after epipolar rectification, and could be an image captured by camera without any processing other than epipolar rectification, or an image obtained by performing certain preprocessing on the image captured by camera. Among them, the preprocessing includes, but is not limited to, one or more of upsampling, pixel merging, smoothing processing, and denoising processing.

[0065] In S102, one original image captured by each camera is respectively encoded into one or more encoded images. The encoding rules may vary according to different application scenarios, but the same encoding rules should be adopted for the original images captured by different cameras, that is, the number of encoded images corresponding to each camera is the same, and for any two original images captured by different cameras in the same exposure, the following should be satisfied: the row coordinates of the encoded pixel rows corresponding to the original pixel rows with the same row coordinates in the two original images are the same.

[0066] Exemplarily, still taking the example of the aforementioned left camera and right camera, if the original image captured by the left camera is encoded into 3 encoded images (hereinafter respectively denoted as the first encoded images 1 - 3), then the original image captured by the right camera will also be encoded into 3 encoded images (hereinafter respectively denoted as the second encoded images 1 - 3).

[0067] If the first row in the left camera image captured by the left camera in the first exposure corresponds to the first row in the first encoded image 1, and corresponds to the second row in the first encoded image 2, and corresponds to the third row in the first encoded image 3, then the first row in the right camera image captured by the right camera in the first exposure should correspond to the first row in the second encoded image 1, and should correspond to the second row in the second encoded image 2, and should correspond to the third row in the second encoded image 3.

[0068] If the first row in the left camera image captured by the left camera in the second exposure corresponds to the second row in the first encoded image 1, and corresponds to the third row in the first encoded image 2, and corresponds to the first row in the first encoded image 3, then the first row in the right camera image captured by the right camera in the first exposure should correspond to the second row in the second encoded image 1, and should correspond to the third row in the second encoded image 2, and should correspond to the first row in the second encoded image 3. And so on.

[0069] Moreover, for each original image captured by the same camera, the following should be satisfied: the size of the encoded image is the same as that of the original image, each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image, and at least two encoded pixel rows correspond to the original pixel rows in different original images, and each encoded pixel row is used to record the pixel values of the corresponding original pixel row.

[0070] Each encoded pixel row can record the pixel values of each pixel point in the corresponding original pixel row, or can only record the pixel values of some pixel points in the corresponding original pixel row, such as only recording the pixel values of the pixel points in a specific area in the corresponding original pixel row.

[0071] In a possible embodiment, to fully increase the density of image information in the encoded image, the original pixel rows corresponding to every two adjacent pixel rows in the encoded image are in different original images. Herein, that the original pixel rows corresponding to each adjacent encoded pixel row are in different original images means that: if any two adjacent encoded pixel rows are selected from the encoded image, the original pixel rows corresponding to these two adjacent encoded pixel rows are in different original images.

[0072] Exemplarily, still taking the foregoing example of the left camera and the right camera, and assuming that the left camera captures left camera images 1-3 respectively during multiple exposures, and the right camera captures right camera images 1-3 respectively during multiple exposures. Since the encoding rules for the left camera images 1-3 are exactly the same as those for the right camera images 1-3, only the encoding of the left camera images 1-3 will be described here.

[0073] If the first row in the left camera image 1 corresponds to the first row in the encoded image, then the second row in the encoded image should correspond to the original pixel row in the left camera image 2 or the left camera image 3. Assuming that the second row in the encoded image corresponds to the second row in the left camera image 2, then the third row in the encoded image should correspond to the original pixel row in the left camera image 1 or the left camera image 3, and so on.

[0074] In other possible embodiments, the original pixel rows corresponding to some adjacent encoded pixel rows in the encoded image may also be in the same original image. Exemplarily, an encoded image includes 10 encoded pixel rows, denoted as encoded pixel rows 1-10 respectively. The original pixel rows corresponding to the encoded pixel rows 1-5 in this encoded image are in the original image 1, and the original pixel rows corresponding to the encoded pixel rows 6-10 in this encoded image are in the original image 2. In this example, some adjacent encoded pixel rows in the encoded image, such as the encoded pixel row 1 and the encoded pixel row 2, the encoded pixel row 7 and the encoded pixel row 8, have corresponding original pixel rows in the same original image.

[0075] Each encoded pixel row in the encoded image corresponds to only one original pixel row, and the length of the encoded pixel row is the same as that of the original pixel row. When recording the image information of the original pixel row, each encoded pixel point in the encoded pixel row is used to record the pixel value of the original pixel point in the corresponding original pixel row that has the same column coordinate as this encoded pixel point. Exemplarily, assuming that the encoded pixel row and the original pixel row include 10 pixel points, then the encoded pixel point with column coordinate 0 in the encoded pixel row is used to record the pixel value of the original pixel point with column coordinate 0 in the original pixel row, and the encoded pixel point with column coordinate 1 in the encoded pixel row is used to record the pixel value of the original pixel point with column coordinate 1 in the original pixel row, and so on.

[0076] The trees in the encoded images obtained by encoding can be different according to different application scenarios. Exemplarily, as Figure 2 described, n is any positive integer greater than 2. For an application scenario with N exposure times, since there are a total of N original images, theoretically, N l -N different encoded images can be obtained by encoding, where l is the number of original pixel rows included in the original image. Taking Figure 2 as an example, Figure 2 in the application scenario shown, if N = 3, then n is a positive integer greater than 2 and less than or equal to 3 l -3.

[0077] Moreover, for the original images captured by different cameras, the same encoding method should be used. Exemplarily, still taking the example of the left and right cameras mentioned above, assuming that only one encoded image is encoded for each camera. If the first row in the encoded image corresponding to the left camera corresponds to the first row in the left camera image 1, then the first row in the encoded image corresponding to the right camera should correspond to the first row in the right camera image 1. If the second row in the encoded image corresponding to the left camera corresponds to the second row in the left camera image 2, then the second row in the encoded image corresponding to the camera should correspond to the second row in the right camera image 2, and so on.

[0078] In S103, the method of matching the encoded images is the same as that of matching the original images. It can be to obtain a disparity image through image matching and then determine the target depth image based on the disparity image, or it can be to directly obtain the target depth image through image matching. Moreover, if each camera corresponds to multiple encoded images, only some of the encoded images can be matched, and all the encoded images can be encoded. The pixel value of each pixel point in the disparity image is used to represent the disparity of the spatial point corresponding to this pixel point in different encoded images, and the pixel value of each pixel point in the depth image is used to represent the depth of the spatial point corresponding to this pixel point.

[0079] Moreover, it can be to only match the encoded images, or to match the encoded images and the original images. For the convenience of description in the following, only the case of only matching the encoded images is taken as an example for illustration. The principle of matching the encoded images and the original images is the same and will not be elaborated here.

[0080] Exemplarily, in a possible embodiment, the aforementioned S103 can be as Figure 3 shown and implemented through the following steps:

[0081] S1031, match the encoded images in each encoded image group to obtain the first image corresponding to the encoded image group.

[0082] Among them, each group of encoded images includes a plurality of the encoded images, and each encoded image corresponds to a different one of the cameras and is obtained in accordance with the same encoding method. The first image is a depth image or a disparity image.

[0083] Still taking the example of the left and right cameras mentioned above, assume that 3 encoded images are obtained by encoding the left camera images 1-3, denoted as the first encoded images 1-3 respectively. Similarly, 3 encoded images are obtained by encoding the right camera images 1-3, denoted as the second encoded images 1-3 respectively.

[0084] Among them, the first encoded image 1 and the second encoded image 1 are encoded in exactly the same encoding method. That is, if the first row in the first encoded image 1 corresponds to the first row in the left camera image 1, then the first row in the second encoded image 1 corresponds to the first row in the right camera image 1. If the second row in the first encoded image 1 corresponds to the second row in the left camera image 2, then the second row in the second encoded image 1 corresponds to the second row in the right camera image 2, and so on. The first encoded image 2 and the second encoded image 2 are encoded in exactly the same encoding method, and the first encoded image 3 and the second encoded image 3 are encoded in exactly the same encoding method.

[0085] In this example, there are three groups of encoded images, namely {the first encoded image 1, the second encoded image 1}, {the first encoded image 2, the second encoded image 2}, {the first encoded image 3, the second encoded image 3}. Therefore, the second encoded images in any one or more of the groups of encoded images can be matched.

[0086] Exemplarily, only the first encoded image 1 and the second encoded image 1 can be matched, or only the first encoded image 2 and the second encoded image 2 can be matched. It is also possible to match the first encoded image 2 and the second encoded image 2, the first encoded image 3 and the second encoded image 3 respectively, or to match the first encoded image 1 and the second encoded image 1, the first encoded image 2 and the second encoded image 2, the first encoded image 3 and the second encoded image 3 respectively. For the case of only one match, compared with performing image matching on the original images captured for each exposure respectively, the number of image matching operations can be effectively reduced, thereby improving the efficiency of generating the depth image. And since the encoded images in the present application contain the image information in each of the original images captured by multiple exposures, even if only one image matching operation is performed, the depth image can be accurately obtained. For the case of multiple matches, since multiple matches are performed, the accuracy of the determined depth image can be further improved.

[0087] S1032, obtaining a target depth image based on one first image, or fusing all the first images to obtain a target depth image.

[0088] In S1031, when each encoded image in an encoded image group is matched, a first image is obtained. Therefore, the number of first images is one or more. The following will be described in different cases according to the number of first images and the categories of the first images:

[0089] Case 1: The number of first images is 1, and the first image is a depth image.

[0090] In this case, the first image is used as the depth image obtained in S1032.

[0091] Case 2: The number of first images is 1, and the first image is a disparity image.

[0092] In this case, depth calculation is performed on the first image to obtain a depth image.

[0093] Case 3: The number of first images is multiple, and the first images are depth images.

[0094] In this case, there are two methods:

[0095] Method 1: Select one first image as the target depth image obtained in S1032.

[0096] Method 2: Fuse all the first images, and use the fusion result as the target depth image.

[0097] Case 4: The number of first images is multiple, and the first images are disparity images.

[0098] In this case, there are two methods:

[0099] Method 3: Select one first image, and perform depth calculation on the selected first image to obtain the target depth image.

[0100] Method 4: Fuse all the first images, and perform depth calculation on the fusion result to obtain the target depth image.

[0101] The following will specifically give an exemplary description of how to encode to obtain the encoded image:

[0102] In a possible embodiment, the steps shown are repeatedly executed Figure 4 until the encoded image is obtained, including:

[0103] S401, for each row coordinate, select a raw pixel row located at the row coordinate from all the raw images captured by the camera.

[0104] When selecting, it can be randomly selected or selected according to a preset rule, such as selecting the original pixel row with the most valid information. Here, the valid information can refer to the valid information of all pixel points in the original pixel row, or it can refer to the valid information of some pixel points in the original pixel row. Exemplarily, if the user is only interested in the area where the column coordinates are in [100, 800], the valid information can refer to the valid information of the pixel points in the original pixel row where the column coordinates are in [100, 800].

[0105] S402, record the image information of the selected original pixel row in the encoded pixel row located at the row coordinate of this row.

[0106] Until all row coordinates are traversed, the construction of the encoded image is completed. The encoded image obtained by encoding can be as Figure 2 shown.

[0107] See Figure 5 , Figure 5 shown is a schematic structural diagram of a depth image acquisition device provided by an embodiment of the present invention, including:

[0108] An original image acquisition module 501, configured to, for each camera, acquire the rectified original image captured by the camera in multiple exposures with different structured lights. Among them, there are at least two cameras, and different cameras are located at different positions;

[0109] An encoding module 502, configured to, according to the same encoding rule, for each camera, encode the original image captured by the camera to obtain at least one encoded image corresponding to the camera. Among them, the size of the encoded image is the same as that of the original image, each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image, and the original pixel rows corresponding to at least two adjacent encoded pixel rows are in different original images. Each encoded pixel row is used to record the pixel value of the corresponding original pixel row;

[0110] A matching module 503, configured to match the encoded images corresponding to all the cameras to obtain a target depth image.

[0111] In a possible embodiment, the matching module 503 matches the encoded images corresponding to all the cameras to obtain a target depth image, including:

[0112] Match the encoded images corresponding to all the cameras and the original images to obtain a target depth image.

[0113] In a possible embodiment, the original pixel rows corresponding to every two adjacent pixel rows in the encoded image are in different original images.

[0114] In a possible embodiment, the matching module 503 matches the encoded images corresponding to all the cameras to obtain a target depth image, including:

[0115] Match the encoded images in at least one encoded image group to obtain a first image corresponding to the encoded image group, where each encoded image group includes a plurality of the encoded images, and different encoded images in each encoded image group correspond to different cameras and are obtained according to the same encoding method, and the first image is a depth image or a disparity image;

[0116] Based on one of the first images, obtain a target depth image; or, fuse all the first images to obtain a target depth image.

[0117] In a possible embodiment, the matching module 503 fuses all the first images to obtain a depth image, including:

[0118] If the first image is a depth image, fuse all the first images to obtain a first fusion result as the target depth image;

[0119] If the first image is a disparity image, fuse all the first images to obtain a second fusion result; perform depth calculation according to the second fusion result to obtain a target depth image.

[0120] In a possible embodiment, the encoding module 502 encodes the original images captured by the cameras to obtain at least one encoded image corresponding to each camera, including:

[0121] Repeat the following steps until an encoded image is obtained:

[0122] For each row coordinate, select an original pixel row located at the row coordinate from all the original images captured by the camera;

[0123] Record the image information of the selected original pixel row in the encoded pixel row located at the row coordinate.

[0124] In a possible embodiment, the encoding module 502 selects an original pixel row located at the row coordinate from all the original images captured by the camera, including:

[0125] Select the original pixel row with the most valid information and located at the row coordinate from all the original images captured by the camera.

[0126] In a possible embodiment, the original image acquisition module 501 acquires, for each camera, the rectified original images captured by the camera in multiple exposures with different structured lights, including:

[0127] For each camera, acquire the images captured by the camera in multiple exposures with different structured lights as the images to be processed;

[0128] For each camera, preprocess the images to be processed captured by the camera to obtain the original images, where the preprocessing includes one or more of upsampling, pixel merging, smoothing processing, and denoising processing.

[0129] An embodiment of the present invention further provides an electronic device, as Figure 6 shown, including a plurality of cameras 601, a plurality of structured light sources 602, a processor 603, and a memory 604.

[0130] The plurality of structured light sources 602 are used to sequentially project structured lights with different spatial distributions for multiple exposures;

[0131] The plurality of cameras 601 are used to respectively capture the original images in the multiple exposures;

[0132] The memory 604 is used to store computer programs;

[0133] The processor 603, when executing the programs stored on the memory 604, implements the following steps:

[0134] For each camera, acquire the rectified original images captured by the camera in multiple exposures with different structured lights, where at least two cameras exist, and different cameras are located at different positions;

[0135] According to the same coding rule, for each camera, encode the original images captured by the camera to obtain at least one encoded image corresponding to the camera, where the size of the encoded image is the same as that of the original image, each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image, and at least two encoded pixel rows correspond to the original pixel rows in different original images, and each encoded pixel row is used to record the pixel values of the corresponding original pixel row;

[0136] Match the encoded images corresponding to all the cameras to obtain the target depth image.

[0137] The memory mentioned in the above electronic device may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0138] The aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0139] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above depth image acquisition methods are implemented.

[0140] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the depth image acquisition methods in the above embodiments.

[0141] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that integrates one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0142] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or device including a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including the element.

[0143] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of the apparatus, electronic device, computer-readable storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0144] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A method for obtaining a depth image, characterized in that The method includes: For each camera, obtaining the rectified original images captured by the camera in multiple exposures with different structured lights, where there are at least two cameras, and different cameras are located at different positions; According to the same encoding rule, for each camera, encoding the original images captured by the camera to obtain at least one encoded image corresponding to the camera, where the size of the encoded image is the same as that of the original image, each encoded pixel row in the encoded image corresponds to the original pixel row with the same row coordinate in the original image, and the original pixel rows corresponding to at least two encoded pixel rows are in different original images, and each encoded pixel row is used to record the pixel values of the corresponding original pixel row; the original pixel rows corresponding to every two adjacent pixel rows in the encoded image are in different original images; Matching the encoded images corresponding to all the cameras to obtain a target depth image.

2. The method according to claim 1, characterized in that, The matching the encoded images corresponding to all the cameras to obtain a target depth image includes: Matching the encoded images in at least one encoded image group to obtain a first image corresponding to the encoded image group, where each encoded image group includes multiple encoded images, and different encoded images in each encoded image group correspond to different cameras and are obtained according to the same encoding method, and the first image is a depth image or a disparity image; Based on one of the first images, obtaining a target depth image; or, fusing all the first images to obtain a target depth image.

3. The method according to claim 2, wherein The fusing all the first images to obtain a depth image includes: If the first image is a depth image, fusing all the first images to obtain a first fusion result as the target depth image; If the first image is a disparity image, fusing all the first images to obtain a second fusion result; performing depth calculation according to the second fusion result to obtain a target depth image.

4. The method according to claim 1, wherein The encoding the original images captured by the camera to obtain at least one encoded image corresponding to the camera includes: Repeatedly executing the following steps until an encoded image is obtained: For each row coordinate, selecting one original pixel row located at the row coordinate from all the original images captured by the camera; Recording the image information of the selected original pixel row in the encoded pixel row located at the row coordinate.

5. The method according to claim 4, wherein The selecting one original pixel row located at the row coordinate from all the original images captured by the camera includes: Selecting the original pixel row with the most valid information and located at the row coordinate from all the original images captured by the camera.

6. The method according to claim 1, wherein The obtaining the rectified original images captured by the camera in multiple exposures with different structured lights for each camera includes: For each camera, obtaining the images captured by the camera in multiple exposures with different structured lights as the images to be processed; For each camera, preprocess the to-be-processed image captured by the camera to obtain a raw image, where the preprocessing includes one or more of upsampling, pixel merging, smoothing processing, and denoising processing.

7. A depth image acquisition device, characterized in that, The device includes: A raw image acquisition module, configured to, for each camera, acquire the rectified raw image captured by the camera in multiple exposures with different structured lights, where there are at least two cameras, and different cameras are located at different positions; An encoding module, configured to encode the raw image captured by the camera according to the same encoding rule to obtain at least one encoded image corresponding to the camera, where the size of the encoded image is the same as that of the raw image, each encoded pixel row in the encoded image corresponds to the raw pixel row with the same row coordinate in the raw image, and the raw pixel rows corresponding to at least two adjacent encoded pixel rows are in different raw images, and each encoded pixel row is used to record the pixel values of the corresponding raw pixel row; the raw pixel rows corresponding to every two adjacent pixel rows in the encoded image are in different raw images; A matching module, configured to match the encoded images corresponding to all the cameras to obtain a target depth image.

8. An electronic device, characterized in that, The electronic device includes multiple cameras, multiple structured light sources, a processor, and a memory; The multiple structured light sources are configured to sequentially project structured lights with different spatial distributions for multiple exposures; The multiple cameras are configured to respectively capture raw images during the multiple exposures; The memory is configured to store a computer program; The processor, when executing the program stored on the memory, implements the method steps according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1-6 are implemented.

Images