Methods, apparatus, equipment, media, and program products for generating stereoscopic images

Abstract

This application discloses a method, apparatus, device, medium, and program product for generating stereoscopic images, belonging to the field of image processing technology. The method includes acquiring a first depth image and a first color image captured simultaneously by an electronic device, wherein the first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device; generating a second depth image based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image; and converting the first color image into a stereoscopic image based on the second depth image and first camera parameters of the first color camera.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a method, apparatus, device, medium, and program product for generating stereoscopic images. Background Technology

[0002] Today, the technology of converting two-dimensional images or videos into three-dimensional images or videos is becoming increasingly widespread. This technology is used in fields such as photography, film, and virtual reality, and can provide users with a more realistic and immersive visual experience.

[0003] Currently, the mainstream technology for converting ordinary 2D (2D) images or videos into 3D (3D) stereoscopic images or videos uses computer vision algorithms to estimate the depth of the 2D image and then renders the 2D image based on the depth estimation result to obtain a 3D stereoscopic image. This method of generating stereoscopic images cannot reproduce true depth information, therefore the visual effect of the stereoscopic images generated by this method is poor. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, medium, and program product for generating stereoscopic images, which can solve the technical problem of poor visual effects in existing stereoscopic images.

[0005] In a first aspect, embodiments of this application provide a method for generating a stereoscopic image, the method comprising:

[0006] Acquire a first depth image and a first color image captured by the electronic device at the same time, wherein the first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device;

[0007] A second depth image is generated based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image;

[0008] Based on the second depth image and the first camera parameters of the first color camera, the first color image is converted into a stereoscopic image.

[0009] Secondly, embodiments of this application provide a stereoscopic image generation apparatus, the apparatus comprising:

[0010] The acquisition module is used to acquire a first depth image and a first color image captured by the electronic device at the same time, wherein the first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device;

[0011] A generation module is configured to generate a second depth image based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image;

[0012] The conversion module is used to convert the first color image into a stereoscopic image based on the second depth image and the first camera parameters of the first color camera.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method provided in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0015] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the method provided in the first aspect.

[0016] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as provided in the first aspect.

[0017] In the stereoscopic image generation method, apparatus, device, medium, and program products of this application, the depth information provided in the first depth image captured by the depth camera has high precision, while the first color image has high resolution. Therefore, the second depth image generated by combining the first depth image and the first color image provides depth information with higher precision and higher resolution. Thus, converting the first color image into a stereoscopic image based on the second depth image can result in a final stereoscopic image with better visual effects and depth perception capabilities. Attached Figure Description

[0018] Figure 1 This is one of the flowcharts illustrating a method for generating a stereoscopic image according to an embodiment of this application;

[0019] Figure 2This is a second schematic flowchart of a method for generating a stereoscopic image provided in another embodiment of this application;

[0020] Figure 3 This is the third flowchart illustrating a method for generating a stereoscopic image according to another embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a stereoscopic image generation apparatus provided in one embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0025] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0026] In existing technologies, the process of converting a planar 2D image into a stereoscopic 3D image involves extracting depth information from the 2D image. Depth information refers to the distance of each pixel from the camera. Then, using the extracted depth information, two 3D images from different perspectives (left-eye image and right-eye image) are synthesized. The visual quality of the 3D image synthesized in this way is relatively poor.

[0027] To address the aforementioned technical problems, this application provides a method for generating stereoscopic images. The method for generating stereoscopic images provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0028] like Figure 1 As shown, Figure 1This is a flowchart illustrating a method for generating a stereoscopic image according to an embodiment of this application. This embodiment provides a method for generating a stereoscopic image, which may include:

[0029] S101, acquire the first depth image and the first color image captured by the electronic device at the same time, wherein the first depth image is a depth image captured by the depth camera on the electronic device, and the first color image is a color image captured by the first color camera on the electronic device;

[0030] In this embodiment, the electronic device includes a depth camera and a first color camera, and can be controlled to capture the same scene at the same time. The depth camera captures a first depth image, and the first color camera captures a first color image.

[0031] The depth camera can be a Time-of-Flight (TOF) camera, and the first color camera can be an RGB camera. A depth image is an image that records the distance information of objects from the observation point. Therefore, a depth image can provide three-dimensional spatial information of each object in the scene. The pixel value of each pixel in the depth image represents the depth value of that pixel, which indicates the distance from the observation point on the depth camera to the corresponding point in the scene. Each pixel in the color image contains the intensity values ​​of three colors: R (red), G (green), and B (blue).

[0032] S102, Generate a second depth image based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image;

[0033] In this embodiment, the resolution of the depth image captured by the depth camera is typically lower than the resolution of the color image. Therefore, the first depth image can be corrected based on the first color image to obtain a second depth image, aligning the first color image and the second depth image in the same coordinate system. Since the resolution of the first depth image captured by the depth camera is lower than the resolution of the first color image, while the resolution of the second depth image is equal to the resolution of the first color image, the resolution of the first depth image is also lower than the resolution of the second depth image. Furthermore, the pixel value of each pixel in the second depth image corresponds to the true depth value of a pixel in the first color image.

[0034] This step generates a high-resolution second depth image based on the low-resolution first depth image and the high-resolution first color image. This second depth image can accurately represent the depth information of each object in the scene, providing a foundation for the subsequent generation of stereo images.

[0035] In some embodiments, S102 includes:

[0036] Determine the first proportion of the number of distant pixels in the first depth image to the total number of pixels in the first depth image. Distant pixels are pixels whose depth value is greater than the depth threshold.

[0037] If the first ratio is greater than the ratio threshold, a second depth image is generated based on the first depth image and the first color image;

[0038] When the first ratio is less than or equal to the ratio threshold, a second depth image is generated based on the first depth image, the first color image, and the second color image. The second color image is a color image captured by a second color camera on an electronic device. The first color image and the second color image are acquired at the same time, and the first color camera and the second color camera have different shooting angles.

[0039] In this embodiment, after obtaining the first depth image, the pixel value corresponding to each pixel in the first depth image can be calculated. Since the pixel value in the first depth image is the depth value, the pixel value of each pixel can be compared with a preset depth threshold.

[0040] For example, the depth threshold can be the limit of depth that binocular stereo vision technology can accurately calculate. Binocular stereo vision technology is a method that uses two or more color cameras to capture images of the same scene from different perspectives to estimate the depth of objects in the scene.

[0041] If the number of distant pixels accounts for a first proportion of the total number of pixels in the first depth image, and this first proportion is greater than a proportion threshold, then it can be assumed that the object in the scene being captured is too far away to be accurately calculated using binocular stereo vision technology. In this case, the depth of the object in the scene can be estimated using only the depth image. There is no need to capture a second color image using a second color camera in the electronic device; the second depth image is generated solely from the first depth image and the first color image.

[0042] If the number of distant pixels accounts for a first proportion of the total number of pixels in the first depth image, and this first proportion is less than or equal to a proportion threshold, then the object being photographed in the scene can be considered relatively close. Binocular stereo vision technology can then be used to accurately calculate the depth of the object in that scene. Therefore, both the depth image and binocular stereo vision technology can be used to estimate the depth of objects in the scene, and these two methods can be combined to generate a high-resolution second depth image. Alternatively, a second color image can be captured by a second color camera in an electronic device, and the second depth image can be generated based on the first depth image, the first color image, and the second color image.

[0043] The percentage threshold can be selected from 60% to 80%, and the depth threshold can be 3 meters, 4 meters, or 5 meters.

[0044] In this embodiment, the above method can determine the depth values ​​of most pixels in the first depth image and select the most suitable depth calculation method to ensure that the generated second depth image is both accurate and stable.

[0045] As an optional embodiment, generating a second depth image based on a first depth image and a first color image includes:

[0046] Obtain the second camera parameters from the depth camera;

[0047] Align the first depth image and the first color image according to the first camera parameters and the second camera parameters to obtain the aligned first depth image and the aligned first color image;

[0048] The second depth image is obtained by performing depth calculations based on the aligned first depth image and the aligned first color image.

[0049] In this embodiment, the first camera parameters include the internal and external parameters of the first color camera, and the second camera parameters include the internal and external parameters of the depth camera. The first and second camera parameters can be used to correct and align image data, aligning the first depth image and the first color image to obtain the second depth image.

[0050] If the first ratio is greater than a ratio threshold, a second depth image can be generated solely from the first depth image and the first color image. For example, the first depth image and the first color image can first be spatially aligned accurately based on first and second camera parameters. Then, depth calculation is performed to obtain the second depth image. This depth calculation can be achieved using traditional algorithms such as bilateral filtering or guided filtering, where the aligned first color image guides the upsampling of the first depth image to obtain a high-resolution second depth image. Alternatively, depth calculation can involve pre-training a neural network model, using the first camera parameters, second camera parameters, the first depth image, and the first color image as inputs. This neural network model can then infer the high-resolution second depth image.

[0051] The second depth image generated in this way can accurately represent the depth information of each object in the scene, providing a foundation for subsequent stereo image generation.

[0052] In some embodiments, generating a second depth image based on a first depth image, a first color image, and a second color image includes:

[0053] Input the first camera parameters, the second camera parameters, the first depth image, and the first color image into the depth inference model to obtain the third depth image output by the depth inference model, and the confidence level of the pixels in the third depth image.

[0054] Generate a disparity image and the confidence level of each pixel in the disparity image based on the first color image and the second color image;

[0055] The parallax image is converted into a fourth depth image, where the pixels in the parallax image correspond one-to-one with the pixels in the fourth depth image, and the confidence level of each pixel in the fourth depth image is equal to the confidence level of the corresponding pixel in the parallax image.

[0056] A second depth image is generated based on the confidence scores of pixels in the third depth image, the confidence scores of pixels in the fourth depth image, and at least one of the third and fourth depth images.

[0057] In this embodiment, when the first ratio is less than or equal to the ratio threshold, the depth of objects in the scene can be estimated using both the depth image and the binocular stereo vision technology, and the two can be combined to generate a high-resolution second depth image.

[0058] Specifically, at the same time, a first color image can be captured by a first color camera and a second color image can be captured by a second color camera in an electronic device. The first and second color cameras have different shooting angles; for example, the first color camera can be a wide-angle camera and the second color camera can be an ultra-wide-angle camera; or, for example, the first color camera can be a wide-angle camera and the second color camera can be a periscope camera.

[0059] Therefore, a neural network model capable of inferring high-resolution depth images, i.e., a deep inference model, can be pre-trained. The inference process of the deep inference model can be as follows: inputting the first camera parameters, the second camera parameters, the first depth image, and the first color image into the deep inference model, the deep inference model uses the weights and features learned through training to analyze the input data, obtaining the third depth image and the first confidence map output by the deep inference model. The first confidence map is used to characterize the reliability of each depth value in the third depth image.

[0060] Similarly, a disparity image can be generated between the first color image and the second color image. The disparity image describes the positional offset of the pixels of the same object in the two color images when viewing the same scene from the perspective of the first color image and the perspective of the second color image. After generating the disparity image, the reliability of each pixel in the disparity image can be further evaluated by calculating the disparity to generate a second confidence map corresponding to the disparity image. This second confidence map is used to characterize the reliability of each disparity value in the disparity image and can convert the disparity image into a fourth depth image. Since the pixels in the disparity image and the pixels in the fourth depth image correspond one-to-one, the second confidence map can also be used to characterize the reliability of each pixel value in the fourth depth image. That is, the confidence of each pixel in the disparity image is equal to the confidence of the corresponding pixel in the fourth depth image.

[0061] In addition, the first color image and the second color image can be aligned using a stereo correction algorithm, and then a stereo matching algorithm can be used to calculate a high-resolution fourth depth image and a second confidence map.

[0062] For example, a stereo correction algorithm might involve: acquiring first calibration data from a first color camera and second calibration data from a second color camera. The calibration data might include focal length, principal point, and distortion coefficients. Then, based on the first and second calibration data, the geometric relationship between the first and second color cameras is calculated. A transformation matrix between the first and second color cameras is then generated based on this geometric relationship. This transformation matrix is ​​used to correct rotational or angular deviations in the viewing angles of the first and second color cameras. Therefore, based on the transformation matrix, identical objects in the first and second color images can be adjusted to align them.

[0063] After obtaining the aligned first and second color images, stereo matching algorithms such as block matching and semi-global matching can be used to process the first and second color images to obtain a disparity image between the first and second color images. Based on the degree of matching between the first and second color images, a second confidence map corresponding to the generated disparity image is determined. Subsequently, the disparity image can be converted into a fourth depth image in the same way. The second confidence map can also be used to characterize the reliability of each pixel value in the fourth depth image.

[0064] After obtaining the third and fourth depth images, the third and fourth depth images can be fused at the pixel level to obtain the second depth image.

[0065] By combining the data from the first depth image with the data captured by the dual color cameras, a high-resolution second depth image can be calculated and fused, thereby improving the accuracy of the second depth image.

[0066] In some embodiments, generating a second depth image based on the confidence levels of pixels in a third depth image, the confidence levels of pixels in a fourth depth image, and at least one of the third and fourth depth images includes:

[0067] For the pixels of the third depth image, perform the first operation to obtain the third depth value corresponding to the pixels of the third depth image;

[0068] Generate a second depth image based on the third depth value;

[0069] The first operation is:

[0070] A first product of a first depth value and a first confidence level is determined, and a second product of a second depth value and a second confidence level is determined, wherein the first depth value is the depth value corresponding to the first pixel, the first confidence level is the confidence level of the first pixel, the second depth value is the depth value corresponding to the second pixel, the second confidence level is the confidence level of the second pixel, the first pixel is any pixel in the third depth image, and the second pixel is the pixel in the fourth depth image corresponding to the first pixel;

[0071] Determine the first sum of the first product and the second product, and the second sum with the first confidence level and the second confidence level;

[0072] Determine the first quotient obtained by dividing the first sum by the second sum;

[0073] Round the first quotient down to obtain the third depth value corresponding to the first pixel.

[0074] In this embodiment, during the pixel-level fusion of the third depth image and the fourth depth image, a first operation can be performed on each pixel in the third depth image. The calculation formula for the first operation is as follows:

[0075]

[0076] Where d1 is the first depth value of the first pixel in the third depth image, d2 is the second depth value of the second pixel in the fourth depth image, c1 is the first confidence level of the first pixel, and c2 is the second confidence level of the second pixel. floor() is the calculation process of rounding down. This is the third depth value.

[0077] After obtaining the third depth values ​​corresponding to the pixels of all third depth images, a second depth image can be generated based on all the third depth values. In the second image, the pixel value of each pixel is a third pixel value.

[0078] In this embodiment, the above method completes the fusion of depth maps. It can consider the confidence level of each depth value and perform fusion by weighted averaging, providing a more balanced and accurate depth estimate under different conditions.

[0079] In some embodiments, generating a second depth image based on the confidence scores of pixels in a third depth image, the confidence scores of pixels in a fourth depth image, the third depth image, and the fourth depth image includes:

[0080] For each pixel in the third depth image, perform the second operation to obtain the third depth value corresponding to the pixel in the third depth image;

[0081] The second depth image is generated based on the third depth value;

[0082] The second operation is as follows:

[0083] If the first confidence level is greater than or equal to the confidence level threshold, the depth value of the first pixel is determined as the third depth value corresponding to the first pixel, wherein the first pixel is any pixel in the third depth image, and the first confidence level is the confidence level of the first pixel.

[0084] If the first confidence level is less than the confidence level threshold, the depth value of the second pixel is determined as the third depth value corresponding to the first pixel, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel.

[0085] In this embodiment, during the pixel-level fusion of the third depth image and the fourth depth image, a second operation can be performed on each pixel of the third depth image. The calculation logic of the second operation can be as follows:

[0086]

[0087] Where d3 is the depth value of the first pixel in the third depth image, d4 is the depth value of the second pixel, and c3 is the first confidence level of the first pixel. This is the third depth value.

[0088] After obtaining the third depth values ​​corresponding to the pixels of all third depth images, a second depth image can be generated based on all the third depth values. In the second image, the pixel value of each pixel is a third pixel value.

[0089] In this embodiment, the above method completes the fusion of depth maps. It can be done intuitively and simply based on confidence level, with clear logic, and can quickly and accurately achieve depth estimation of images.

[0090] S103, convert the first color image into a stereo image based on the second depth image and the first camera parameters of the first color camera.

[0091] In this embodiment, the first camera parameters of the first color camera include the internal and external parameters of the first color camera. Based on the first camera parameters and the second depth image, the coordinates of each pixel in the first color image can be transformed from the first image coordinate system to the world coordinate system. Then, the pixel values ​​in the world coordinate system are projected onto the second image coordinate system, thus obtaining two new stereoscopic images.

[0092] Specifically, during the coordinate system transformation, the z-axis coordinates of each pixel in the first color image in the world coordinate system can be determined based on the depth value in the second depth image, and the x-axis and y-axis coordinates of each pixel in the first color image in the world coordinate system can be determined based on the coordinates of the pixel in the first image coordinate system.

[0093] Two stereoscopic images can also be generated by translating the first color image, resulting in a left-eye stereoscopic image and a right-eye stereoscopic image. Specifically, the left-eye view image is generated by translating the pixels of the first color image in the negative direction based on the second depth image, and the right-eye stereoscopic image is generated by translating the pixels of the first color image in the positive direction based on the second depth image. For example, closer objects with smaller depth values ​​can be translated more, while distant objects with larger depth values ​​can be translated less.

[0094] After obtaining the stereo image, if there are holes and uneven areas due to foreground and background occlusion, etc., image inpainting algorithms using traditional or neural network models can be used to fill the holes; for uneven areas, image filtering algorithms can be used to smooth the image.

[0095] As an alternative embodiment, such as Figure 2 As shown, the method for generating stereo images can be applied to mobile phones, which include a depth camera. After capturing the original depth image using the depth camera, step S201 can be further executed: determining the calculation method for the target depth image. Here, the original depth image is a low-resolution depth image with a resolution lower than required, and the target depth image is a high-resolution depth image with the required resolution.

[0096] In determining which calculation method to use to calculate the high-resolution target depth image, it can be determined whether the first proportion of pixels with depth values ​​greater than d in the original depth image exceeds a preset proportion threshold, where d is the maximum depth distance that the dual-camera color camera can accurately measure. If the first proportion exceeds the proportion threshold, then S202 is executed: depth calculation is performed using only TOF data to obtain the target depth image. If the first proportion is less than or equal to the proportion threshold, then S203 is executed: depth map calculation is performed using both TOF data and dual-camera RGB data to obtain the target depth image. TOF (Time-of-Flight) data is depth information obtained from the original depth image using time-of-flight measurement technology. Dual-camera RGB data consists of the RGB data of two color images of the same scene captured by the two RGB cameras in the phone, and the scene captured is the same as the scene captured by the original depth image. RGB data refers to the intensity values ​​of the red (R), green (G), and blue (B) color channels of each pixel in the color image. These intensity values ​​are typically in the range of 0 to 255, and by combining these three values, the color of each pixel in the image can be represented.

[0097] After obtaining the high-resolution target depth image, step S204 is executed: based on the original depth image and the target depth image, a new perspective image conforming to the human eye baseline is generated. The new perspective image can be used for 3D display or as a stereoscopic image displayed in a stereoscopic vision system.

[0098] After obtaining the new perspective image, the generated new perspective image may have some defects, such as holes or uneven areas caused by occlusion of foreground and background objects. Therefore, the system can perform S205: post-processing of the new perspective image, which may include hole filling and image smoothing to improve image quality.

[0099] Furthermore, as another optional embodiment, in the process of calculating the depth map using TOF data and dual-camera RGB data to obtain the target depth image, the specific steps can be as follows: Figure 3 The target depth image is calculated in the manner shown.

[0100] First, execute S301: use TOF data and single-camera RGB data to perform depth calculation to obtain the depth. Figure 1 and confidence level Figure 1 The single-camera RGB data refers to the RGB data of a color image of the same scene captured by one of the phone's RGB cameras and the original depth image. The TOF data and the single-camera RGB data can be input into a pre-trained neural network model to obtain the depth image output by the neural network model. Figure 1 and confidence level Figure 1 .

[0101] Then, S302 can be executed: use the dual-camera RGB data to perform depth calculation and obtain the depth. Figure 2 and confidence level Figure 2 Specifically, a stereo matching algorithm can be used to obtain a high-resolution disparity map and confidence level based on dual-camera RGB images. Figure 2 Finally, the high-resolution disparity map is converted to depth. Figure 2 .

[0102] Finally, execute S303 to adjust the depth. Figure 1 and depth Figure 2 Pixel-level fusion is performed to obtain a high-resolution target depth image.

[0103] In this application, the first depth image captured by the depth camera provides high-precision depth information, while the first color image has high resolution. Therefore, the second depth image generated by combining the first depth image and the first color image provides higher-precision and higher-resolution depth information. Thus, converting the first color image into a stereo image based on the second depth image can result in a stereo image with better visual effects and depth perception capabilities.

[0104] Figure 4 This is a schematic diagram of the structure of a stereoscopic image generation device provided in another embodiment of this application, as shown below. Figure 4 As shown, the apparatus for generating the stereoscopic image may include:

[0105] The acquisition module 401 is used to acquire a first depth image and a first color image captured by the electronic device at the same time, wherein the first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device;

[0106] The generation module 402 is used to generate a second depth image based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image;

[0107] The conversion module 403 is used to convert the first color image into a stereoscopic image based on the second depth image and the first camera parameters of the first color camera.

[0108] In this application, the first depth image captured by the depth camera provides high-precision depth information, while the first color image has high resolution. Therefore, the second depth image generated by combining the first depth image and the first color image provides higher-precision and higher-resolution depth information. Thus, converting the first color image into a stereo image based on the second depth image can result in a stereo image with better visual effects and depth perception capabilities.

[0109] In another alternative example, generation module 402 includes:

[0110] The determining unit is used to determine the first proportion of the number of distant pixels in the first depth image to the total number of pixels in the first depth image, where distant pixels are pixels with a depth value greater than a depth threshold.

[0111] The first generation unit is configured to generate a second depth image based on the first depth image and the first color image when the first ratio is greater than a ratio threshold.

[0112] The second generation unit is used to generate a second depth image based on the first depth image, the first color image, and the second color image when the first ratio is less than or equal to the ratio threshold. The second color image is a color image captured by a second color camera on an electronic device. The first color image and the second color image are acquired at the same time, and the first color camera and the second color camera have different shooting angles.

[0113] In another alternative example, the first generating unit is also used for:

[0114] Obtain the second camera parameters from the depth camera;

[0115] Align the first depth image and the first color image according to the first camera parameters and the second camera parameters to obtain the aligned first depth image and the aligned first color image;

[0116] The second depth image is obtained by performing depth calculations based on the aligned first depth image and the aligned first color image.

[0117] In another alternative example, the second generating unit is also used for:

[0118] Input the first camera parameters, the second camera parameters, the first depth image, and the first color image into the depth inference model to obtain the third depth image output by the depth inference model, and the confidence level of the pixels in the third depth image.

[0119] Generate a disparity image and the confidence level of each pixel in the disparity image based on the first color image and the second color image;

[0120] The parallax image is converted into a fourth depth image, where the pixels of the parallax image correspond one-to-one with the pixels of the fourth depth image, and the confidence level of each pixel in the fourth depth image is equal to the confidence level of the corresponding pixel in the parallax image.

[0121] A second depth image is generated based on the confidence scores of pixels in the third depth image, the confidence scores of pixels in the fourth depth image, and at least one of the third and fourth depth images.

[0122] In another alternative example, the second generating unit is also used for:

[0123] For each pixel in the third depth image, perform the first operation to obtain the third depth value corresponding to the pixel in the third depth image;

[0124] Generate a second depth image based on the third depth value;

[0125] The first operation is:

[0126] Determine the first product of the first depth value and the first confidence level, and the second product of the second depth value and the second confidence level, wherein the first depth value is the depth value corresponding to the first pixel, the first confidence level is the confidence level of the first pixel, the second depth value is the depth value corresponding to the second pixel, the second confidence level is the confidence level of the second pixel, the first pixel is any pixel in the third depth image, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel;

[0127] Determine the first sum of the first product and the second product, and the second sum with the first confidence level and the second confidence level;

[0128] Determine the first quotient obtained by dividing the first sum by the second sum;

[0129] Round the first quotient down to obtain the third depth value corresponding to the first pixel.

[0130] In another alternative example, the second generating unit is also used for:

[0131] For each pixel in the third depth image, perform the second operation to obtain the third depth value corresponding to the pixel in the third depth image;

[0132] Generate a second depth image based on the third depth value;

[0133] The second operation is as follows:

[0134] If the first confidence level is greater than or equal to the confidence level threshold, the depth value of the first pixel is determined as the third depth value corresponding to the first pixel, where the first pixel is any pixel in the third depth image, and the first confidence level is the confidence level of the first pixel.

[0135] If the first confidence level is less than the confidence level threshold, the depth value of the second pixel is determined as the third depth value corresponding to the first pixel, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel.

[0136] The stereoscopic image generation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device to these types of devices.

[0137] The stereoscopic image generation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0138] The stereoscopic image generation apparatus provided in this application embodiment can achieve Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0139] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 100, including a processor 110, a memory 119, and a program or instructions stored in the memory 119 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described stereoscopic image generation method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0140] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0141] Please refer to the following: Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 100 includes, but is not limited to, components such as: a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0142] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 120 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0143] The input unit 124 is used to acquire a first depth image and a first color image captured by the electronic device at the same time. The first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device.

[0144] Processor 120 is configured to generate a second depth image based on a first depth image and a first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image;

[0145] The processor 120 is used to convert the first color image into a stereoscopic image based on the second depth image and the first camera parameters of the first color camera.

[0146] In this application, the first depth image captured by the depth camera provides high-precision depth information, while the first color image has high resolution. Therefore, the second depth image generated by combining the first depth image and the first color image provides higher-precision and higher-resolution depth information. Thus, converting the first color image into a stereo image based on the second depth image can result in a stereo image with better visual effects and depth perception capabilities.

[0147] In another alternative example, the processor 120 is used to determine a first proportion of the number of distant pixels in the first depth image to the total number of pixels in the first depth image, where distant pixels are pixels with depth values ​​greater than a depth threshold.

[0148] If the first ratio is greater than the ratio threshold, a second depth image is generated based on the first depth image and the first color image;

[0149] When the first ratio is less than or equal to the ratio threshold, a second depth image is generated based on the first depth image, the first color image, and the second color image. The second color image is a color image captured by a second color camera on an electronic device. The first color image and the second color image are acquired at the same time, and the first color camera and the second color camera have different shooting angles.

[0150] In another alternative example, processor 120 is also used for:

[0151] Obtain the second camera parameters from the depth camera;

[0152] Align the first depth image and the first color image according to the first camera parameters and the second camera parameters to obtain the aligned first depth image and the aligned first color image;

[0153] The second depth image is obtained by performing depth calculations based on the aligned first depth image and the aligned first color image.

[0154] In another alternative example, processor 120 is also used for:

[0155] Input the first camera parameters, the second camera parameters, the first depth image, and the first color image into the depth inference model to obtain the third depth image output by the depth inference model, and the confidence level of the pixels in the third depth image.

[0156] Generate a disparity image and the confidence level of each pixel in the disparity image based on the first color image and the second color image;

[0157] The parallax image is converted into a fourth depth image, where the pixels of the parallax image correspond one-to-one with the pixels of the fourth depth image, and the confidence level of each pixel in the fourth depth image is equal to the confidence level of the corresponding pixel in the parallax image.

[0158] A second depth image is generated based on the confidence scores of pixels in the third depth image, the confidence scores of pixels in the fourth depth image, and at least one of the third and fourth depth images.

[0159] In another alternative example, processor 120 is also used for:

[0160] For each pixel in the third depth image, perform the first operation to obtain the third depth value corresponding to the pixel in the third depth image;

[0161] Generate a second depth image based on the third depth value;

[0162] The first operation is:

[0163] Determine the first product of the first depth value and the first confidence level, and the second product of the second depth value and the second confidence level, wherein the first depth value is the depth value corresponding to the first pixel, the first confidence level is the confidence level of the first pixel, the second depth value is the depth value corresponding to the second pixel, the second confidence level is the confidence level of the second pixel, the first pixel is any pixel in the third depth image, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel;

[0164] Determine the first sum of the first product and the second product, and the second sum with the first confidence level and the second confidence level;

[0165] Determine the first quotient obtained by dividing the first sum by the second sum;

[0166] Round the first quotient down to obtain the third depth value corresponding to the first pixel.

[0167] In another alternative example, processor 120 is also used for:

[0168] For each pixel in the third depth image, perform the second operation to obtain the third depth value corresponding to the pixel in the third depth image;

[0169] Generate a second depth image based on the third depth value;

[0170] The second operation is as follows:

[0171] If the first confidence level is greater than or equal to the confidence level threshold, the depth value of the first pixel is determined as the third depth value corresponding to the first pixel, where the first pixel is any pixel in the third depth image, and the first confidence level is the confidence level of the first pixel.

[0172] If the first confidence level is less than the confidence level threshold, the depth value of the second pixel is determined as the third depth value corresponding to the first pixel, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel.

[0173] It should be understood that, in this embodiment, the input unit 124 may include a graphics processing unit (GPU) 1241 and a microphone 1242. The GPU 1241 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 126 may include a display panel 1261, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 127 includes at least one of a touch panel 1271 and other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0174] The memory 129 can be used to store software programs and various data. The memory 129 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 129 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 129 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0175] Processor 120 may include one or more processing units; optionally, processor 120 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 120.

[0176] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described method for generating stereoscopic images and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0177] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0178] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described stereoscopic image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0179] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0180] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described method for generating stereoscopic images, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0183] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for generating a stereoscopic image, characterized in that, The method is performed by an electronic device and includes: Acquire a first depth image and a first color image captured by the electronic device at the same time, wherein the first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device, and the resolution of the first depth image is smaller than the resolution of the first color image; A second depth image is generated based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image; Based on the second depth image and the first camera parameters of the first color camera, the first color image is converted into a stereoscopic image; The step of generating a second depth image based on the first depth image and the first color image includes: Determine a first proportion of the number of distant pixels in the first depth image to the total number of pixels in the first depth image, wherein the distant pixels are pixels with a depth value greater than a depth threshold; If the first ratio is greater than the ratio threshold, the second depth image is generated based on the first depth image and the first color image; When the first ratio is less than or equal to the ratio threshold, the first camera parameters, the second camera parameters, the first depth image and the first color image are input into the depth inference model to obtain the third depth image output by the depth inference model and the confidence of the pixels of the third depth image. The second camera parameters are the camera parameters of the depth camera. The depth inference model is capable of using the weights and features learned by it through training to analyze the input data to infer a high-resolution depth image. A parallax image and the confidence level of the pixels in the parallax image are generated based on the first color image and the second color image. The second color image is a color image captured by the second color camera on the electronic device. The first color image and the second color image are acquired at the same time. The first color camera and the second color camera have different shooting angles. The resolution of the first depth image is smaller than the resolution of the second color image. The parallax image is converted into a fourth depth image, wherein the pixels of the parallax image and the pixels of the fourth depth image correspond one-to-one, and the confidence level of each pixel of the fourth depth image is equal to the confidence level of the corresponding pixel of the parallax image. The second depth image is generated based on at least one of the third depth image and the fourth depth image, as well as the confidence scores of the pixels in the third depth image and the pixel confidence scores of the fourth depth image.

2. The method according to claim 1, characterized in that, The step of generating the second depth image based on the first depth image and the first color image includes: Obtain the second camera parameters of the depth camera; Align the first depth image and the first color image according to the first camera parameters and the second camera parameters to obtain the aligned first depth image and the aligned first color image; The second depth image is obtained by performing depth calculation based on the aligned first depth image and the aligned first color image.

3. The method according to claim 1, characterized in that, The step of generating the second depth image based on at least one of the third depth image and the fourth depth image, as well as the confidence scores of the pixels in the third depth image and the pixel confidence scores in the fourth depth image, includes: For the pixels of the third depth image, perform the first operation to obtain the third depth value corresponding to the pixels of the third depth image; The second depth image is generated based on the third depth value; The first operation is as follows: A first product of a first depth value and a first confidence level is determined, as well as a second product of a second depth value and a second confidence level, wherein the first depth value is the depth value corresponding to the first pixel, the first confidence level is the confidence level of the first pixel, the second depth value is the depth value corresponding to the second pixel, the second confidence level is the confidence level of the second pixel, the first pixel is any pixel in the third depth image, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel. Determine a first sum of the first product and the second product, and a second sum of the first confidence level and the second confidence level; Determine the first quotient obtained by dividing the first sum by the second sum; The first quotient is rounded down to obtain the third depth value corresponding to the first pixel.

4. The method according to claim 1, characterized in that, The step of generating the second depth image based on at least one of the third depth image and the fourth depth image, as well as the confidence scores of the pixels in the third depth image and the pixel confidence scores in the fourth depth image, includes: For the pixels of the third depth image, perform the second operation to obtain the third depth value corresponding to the pixels of the third depth image; The second depth image is generated based on the third depth value; The second operation is as follows: If the first confidence level is greater than or equal to the confidence level threshold, the depth value of the first pixel is determined as the third depth value corresponding to the first pixel, wherein the first pixel is any pixel in the third depth image, and the first confidence level is the confidence level of the first pixel. If the first confidence level is less than the confidence level threshold, the depth value of the second pixel is determined as the third depth value corresponding to the first pixel, and the second pixel is the pixel in the fourth depth image that corresponds to the first pixel.

5. A device for generating a stereoscopic image, characterized in that, include: The acquisition module is used to acquire a first depth image and a first color image captured by the electronic device at the same time. The first depth image is a depth image captured by a depth camera on the electronic device, and the first color image is a color image captured by a first color camera on the electronic device. The resolution of the first depth image is smaller than the resolution of the first color image. A generation module is configured to generate a second depth image based on the first depth image and the first color image, wherein the resolution of the second depth image is equal to the resolution of the first color image; The conversion module is used to convert the first color image into a stereoscopic image based on the second depth image and the first camera parameters of the first color camera; The generation module is used for: The determining unit is used to determine a first ratio of the number of distant pixels in the first depth image to the total number of pixels in the first depth image, wherein the distant pixels are pixels with a depth value greater than a depth threshold. The first generation unit is configured to generate the second depth image based on the first depth image and the first color image when the first ratio is greater than the ratio threshold. The second generation unit is used to input the first camera parameters, the second camera parameters, the first depth image, and the first color image into a depth inference model when the first ratio is less than or equal to the ratio threshold, to obtain a third depth image output by the depth inference model and the confidence level of the pixels of the third depth image. The second camera parameters are the camera parameters of the depth camera. The depth inference model is capable of using the weights and features learned by it through training to analyze the input data to infer a high-resolution depth image. A parallax image and the confidence level of the pixels in the parallax image are generated based on the first color image and the second color image. The second color image is a color image captured by the second color camera on the electronic device. The first color image and the second color image are acquired at the same time. The first color camera and the second color camera have different shooting angles. The resolution of the first depth image is smaller than the resolution of the second color image. The parallax image is converted into a fourth depth image, wherein the pixels of the parallax image and the pixels of the fourth depth image correspond one-to-one, and the confidence level of each pixel of the fourth depth image is equal to the confidence level of the corresponding pixel of the parallax image. The second depth image is generated based on at least one of the third depth image and the fourth depth image, as well as the confidence scores of the pixels in the third depth image and the pixel confidence scores of the fourth depth image.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method for generating a stereoscopic image as described in any one of claims 1-4.

7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method for generating a stereoscopic image as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps of the method for generating a stereoscopic image as described in any one of claims 1-4.

Images