Method and device for generating binocular picture based on monocular picture

Abstract

The invention discloses a method and device for generating a binocular picture based on a monocular picture, and belongs to the technical field of image processing, and the method comprises the following steps: obtaining an original monocular picture, and carrying out the monocular depth prediction of the original monocular picture to obtain a foreground depth map; respectively generating a mask depth map and a background depth map according to the foreground depth map, and performing hybrid calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map; and obtaining distortion change values corresponding to the left eye view and the right eye view, and deforming the original monocular picture according to the distortion change values and the target depth map to obtain a binocular picture corresponding to the original monocular picture. According to the method, the three-dimensional picture can be directly operated and displayed on equipment such as an iPhone, an iPad, a Mac and a VisionOS, a user can quickly convert materials shot or recorded in the past into the three-dimensional picture and check the three-dimensional picture, the method is simple and quick, and the user experience is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for generating a binocular image based on a monocular image. Background Art

[0002] With the surge in market demand for stereoscopic effects and the frequent updates and iterations of spatial devices such as virtual reality (VR) and mixed reality (MR), ordinary images shot or recorded by traditional monocular cameras can no longer meet users' needs for immersive experience in these devices. Although stereoscopic images can significantly enhance users' visual experience, they are not widely used due to the limitations of existing hardware devices.

[0003] Currently, there are some solutions on the market that try to solve this problem. For example, the Vision Pro device already has the function of converting ordinary photos into photos with a sense of spatial depth in its unofficial VisionOS 2.0 Beta version. However, this function only applies to photos and is limited to specific operating system versions and device types. Devices such as iPhone, iPad and Mac do not support this function.

[0004] In addition, most technical solutions on the market rely on cloud servers to perform complex image processing tasks, which means that data needs to be uploaded to the server to complete the conversion and finally downloaded back to the client device. This method not only increases network latency and affects the user's real-time experience, but also poses challenges to user privacy protection. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for generating a binocular image based on a monocular image, so as to solve the problem in the prior art that the conversion of a monocular image to a binocular image is limited by the operating system version and the device type and relies on a cloud server for image processing.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] A method for generating a binocular image based on a monocular image in the present application comprises the following steps:

[0008] Acquire an original monocular picture, and perform monocular depth prediction on the original monocular picture to obtain a foreground depth map;

[0009] Generate a mask depth map and a background depth map according to the foreground depth map, and perform mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map;

[0010] The distortion change values ​​corresponding to the left and right eye views are obtained respectively, and the original monocular image is deformed according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image.

[0011] Preferably, generating a mask depth map according to the foreground depth map comprises:

[0012] Processing the foreground depth map using an edge detection algorithm to obtain an edge detection image, and determining whether the edge detection image is generated normally;

[0013] When the edge detection image is generated normally, converting the edge detection image into a binary image using a color threshold algorithm, and determining whether the binary image is generated normally;

[0014] When the binary image is generated normally, smoothing the binary image using a fuzzy algorithm to obtain a smoothed binary image, and determining whether the smoothed binary image is generated normally;

[0015] When the smoothed binary image is generated normally, a color threshold algorithm is used to perform background extraction on the smoothed binary image to obtain a mask depth map.

[0016] Preferably, generating a background depth map according to the foreground depth map comprises:

[0017] The foreground depth map is smoothed using a fuzzy algorithm to obtain a background depth map.

[0018] Preferably, the step of performing mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain the target depth map comprises:

[0019] The foreground depth map, the mask depth map and the background depth map are sampled respectively, and all the sampling results are mixed and calculated according to the mask value to output a target depth map.

[0020] Preferably, the obtaining of the distortion change values ​​corresponding to the left and right eye views respectively, and deforming the original monocular image according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image, comprises:

[0021] Sampling the target depth map to obtain a first sampling result, and sampling the original monocular image according to a first channel value of the first sampling result to obtain a second sampling result;

[0022] Calculate the distortion change values ​​corresponding to the left and right eye views respectively according to the focal length and interocular distance of the camera and the depth information of the second sampling result;

[0023] Performing matrix transformation on the original monocular image according to the distortion change values ​​corresponding to the left and right eye views, respectively, to obtain left and right eye views;

[0024] The transparency in the left and right eye views is extracted respectively, and the transparency is optimized using a transparency optimization algorithm to obtain a binocular image.

[0025] Preferably, the first channel value is depth information.

[0026] Preferably, performing matrix transformation on the original monocular image according to the distortion change values ​​respectively corresponding to the left and right eye views to obtain the left and right eye views respectively comprises:

[0027] Performing matrix transformation on the original monocular image according to the distortion change values ​​corresponding to the left and right eye views, respectively, to obtain two sets of new pixel positions, and determining whether the two sets of new pixel positions are both beyond the image range;

[0028] When the two sets of new pixel positions are within the image range, calculating the offset between the second sampling result and the two sets of new pixel positions;

[0029] Creating a grid including all pixel coordinates in the second sampling result, and adding the offsets to the corresponding coordinates in the grid to obtain corresponding new coordinates of the left and right views;

[0030] The second sampling results are remapped according to the new coordinates to obtain left-eye and right-eye views.

[0031] A device for generating a binocular image based on a monocular image, comprising:

[0032] A prediction module, used for acquiring an original monocular picture, and performing monocular depth prediction on the original monocular picture to obtain a foreground depth map;

[0033] A conversion module, used to generate a mask depth map and a background depth map according to the foreground depth map, and perform mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map;

[0034] The distortion module is used to obtain the distortion change values ​​corresponding to the left and right eye views respectively, and deform the original monocular image according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image.

[0035] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for generating a binocular image based on a monocular image as described in any one of the above.

[0036] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement any one of the above-described methods for generating a binocular image based on a monocular image when executed.

[0037] This application has the following beneficial effects:

[0038] This application realizes the direct operation and display of stereoscopic images on devices such as iPhone, iPad, Mac and VisionOS. Through this method, users can also quickly convert previously shot or recorded materials into stereoscopic images and view them; at the same time, based on the player application, users can directly play ordinary videos and instantly render stereoscopic images through this method; based on the shooting application, users can also realize stereoscopic image shooting and recording on devices that originally do not support stereoscopic image shooting; in addition, this method also supports the three-dimensional upgrade of user materials through applications such as network disk, thereby significantly improving the user experience in spatial devices such as VR and MR. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of a method for generating a binocular image based on a monocular image provided by an embodiment of the present application;

[0040] Figure 2 1 is a schematic diagram of a structure of a device for generating a binocular image based on a monocular image provided by an embodiment of the present application;

[0041] Figure 3 It is a schematic diagram of an electronic device for implementing a method for generating a binocular image based on a monocular image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] To make the technical solution of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The terms "first", "second", etc. in the claims and specification of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is only to describe the distinction method used when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that a process, method, system, product or device containing a series of units is not necessarily limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] like Figure 1 As shown, this embodiment provides a method for generating a binocular image based on a monocular image, comprising the following steps:

[0044] S110, obtaining an original monocular image, and performing monocular depth prediction on the original monocular image to obtain a foreground depth map;

[0045] S120, generating a mask depth map and a background depth map respectively according to the foreground depth map, and performing a mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map;

[0046] S130 , obtaining distortion change values ​​corresponding to the left and right eye views respectively, and deforming the original monocular image according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image.

[0047] The method proposed in this embodiment is mainly used in iOS, iPadOS, macOS, VisionOS, Apple-related application ecosystems, VR and MR-related space devices or servers, and is used to convert ordinary images into stereoscopic binocular images, that is, 2D to 3D, but not 3D modeling.

[0048] Specifically, the original monocular image to be processed is obtained, and a monocular depth prediction is performed on it through an AI model. The AI ​​model here can be a monocular depth estimation model Monocular Depth Estimation, or other open source models that can achieve the same effect, to obtain a foreground depth map.

[0049] Further, the foreground depth map is processed using an edge detection algorithm to obtain an edge detection image, and it is determined whether the edge detection image is generated normally;

[0050] When the edge detection image is generated normally, the edge detection image is converted into a binary image using a color threshold algorithm, and it is determined whether the binary image is generated normally;

[0051] When the binary image is generated normally, a fuzzy algorithm is used to smooth the binary image to obtain a smoothed binary image, and whether the smoothed binary image is generated normally is determined;

[0052] When the smoothed binary image is generated normally, a color threshold algorithm is used to extract the background of the smoothed binary image to obtain a mask depth map.

[0053] The foreground depth map is processed by an edge detection algorithm to obtain an edge detection image to highlight the edge of the depth information and determine whether the edge detection image is generated normally. The judgment is based on whether the output image is normal, such as whether the image is empty, whether the encoding format of the image is supported, and whether the output is empty. The specific adjustment can be made according to the actual online operating environment. If normal, the edge detection image is binary-divided using a color threshold algorithm, that is, pixels in the edge detection image with brightness higher than the set threshold are set to one color, and pixels with brightness lower than the set threshold are set to another color. In this way, the colors in the edge detection image can be divided into two extremes, thereby obtaining a high-contrast image. A binary image of degree is obtained, and it is determined whether the binary image is generated normally. If so, a fuzzy algorithm is used to smooth the binary image to obtain a binary image with a smooth and soft visual effect. The fuzzy algorithm used here can be a Gaussian fuzzy algorithm or other fuzzy algorithms with similar effects. At the same time, it is determined whether the smoothed binary image is generated normally. If so, a color threshold algorithm is used to extract the background. This algorithm can not only redefine the boundary, but also create a smooth but clear outline to obtain a mask depth map. At the same time, it is determined whether the mask depth map is generated normally. If so, the final mask depth map is output.

[0054] Next, the foreground depth map is processed using a fuzzy algorithm to obtain a background depth map with a soft and smooth visual effect.

[0055] Among them, the foreground depth map mainly focuses on the foreground objects in the scene, that is, the objects closest to the camera or observer. It provides the distance information between the foreground objects and the camera and is usually used to highlight or analyze the main elements in the scene.

[0056] The ‌Background Depth Map‌ focuses on the background part of the scene, that is, the objects behind the foreground. It provides distance information between the background objects and the camera, which helps to understand the overall structure of the scene and the position of the background elements.

[0057] ‌Mask depth map‌, also known as depth mask, is a special depth map that creates a mask by assigning a depth value to each pixel. This mask can be used to block or hide certain parts of the scene to achieve specific visual effects or analysis purposes. Mask depth maps are widely used in computer graphics, virtual reality, and augmented reality. For example, the mask depth map can be adjusted to control which parts of the information should be displayed or hidden, thereby achieving more complex image processing effects.

[0058] Then, the obtained foreground depth map, mask depth map and background depth map are mixed. Specifically, these images are sampled respectively, and then all the sampling results are mixed and calculated according to the mask value, and finally the result is output to obtain the target depth map.

[0059] Further, the target depth map is sampled to obtain a first sampling result, and the original monocular image is sampled according to the first channel value of the first sampling result to obtain a second sampling result;

[0060] Calculate the distortion change values ​​corresponding to the left and right eye views respectively according to the focal length and interocular distance of the camera and the depth information of the second sampling result;

[0061] According to the distortion change values ​​corresponding to the left and right eye views, the original monocular image is matrix-transformed to obtain the left and right eye views;

[0062] The transparency in the left and right eye views is extracted respectively, and the transparency is optimized using a transparency optimization algorithm to obtain a binocular image.

[0063] First, the target depth map is sampled, and then the first channel value of the sampled image, i.e., the first sampling result, is extracted. Usually, the depth map is one-dimensional, and each pixel has only one value to represent the depth, but it can also be multi-channel. In this case, only one channel carries the actual depth information, and the other channels may be retained for compatibility with certain image processing libraries or formats. When the first channel is mentioned, it usually refers to the single channel that stores the depth information in the depth map. Therefore, the first channel value extracted here is actually the depth information, and the original monocular image is sampled according to its depth information, while ensuring that the sampled original monocular image, i.e., the second sampling result, is consistent with the sampled target depth map size. Then, the disparity of each pixel relative to the central viewpoint, i.e., the horizontal displacement, is calculated based on the focal length of the camera, the interocular distance, and the depth information of the second sampling result. This displacement value is the distortion change value. For the left eye view, the distortion change value is a negative value (offset to the left), and for the right eye view, the distortion change value is a positive value (offset to the right). After all the calculations are completed, the corresponding values ​​of the left and right views are calculated. The distortion change value is used to perform matrix transformation on the original monocular image to change the position of the pixels in the original monocular image, thereby obtaining two sets of new pixel positions corresponding to the left and right views respectively, and the two sets of new pixel positions are compared with the image range respectively. If the two sets of new pixel positions do not exceed the image range, the offset between the second sampling result and the two sets of new pixel positions is calculated, and a grid containing all pixel coordinates in the second sampling result is created at the same time, and these offsets are added to the corresponding coordinates in the grid to obtain new coordinates of each pixel corresponding to the left and right views, and then the second sampling result is remapped according to these new coordinates to obtain the left and right eye views, and then the transparency Alpha in the left and right eye views is extracted respectively, and the transparency optimization algorithm is used to optimize these transparencies respectively. Specifically, four sampling directions are defined: up, down, left, and right. For each pixel, the sampling point positions in the four directions around it are calculated, and the Alpha of the four sampling points is obtained and compared, and the maximum value among them is taken as the final transparency value of the pixel, thereby obtaining the optimized left and right eye views, that is, the binocular image.

[0064] This embodiment realizes the direct operation and display of stereoscopic images on devices such as iPhone, iPad, Mac and VisionOS. Through this method, users can also quickly convert previously shot or recorded materials into stereoscopic images and view them; at the same time, based on the player application, users can directly play ordinary videos and instantly render stereoscopic images through this method; based on the shooting application, users can also realize stereoscopic image shooting and recording on devices that originally do not support stereoscopic image shooting; in addition, this method also supports the three-dimensional upgrade of user materials through applications such as network disk, thereby significantly improving the user experience in spatial devices such as VR and MR.

[0065] like Figure 2 As shown, this embodiment also provides a device for generating a binocular image based on a monocular image, including:

[0066] The prediction module 10 is used to obtain an original monocular picture and perform monocular depth prediction on the original monocular picture to obtain a foreground depth map;

[0067] The conversion module 20 is used to generate a mask depth map and a background depth map according to the foreground depth map, and perform mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map;

[0068] The distortion module 30 is used to obtain the distortion change values ​​corresponding to the left and right eye views respectively, and deform the original monocular image according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image.

[0069] An implementation manner of the above-mentioned device may be: the prediction module 10 obtains the original monocular picture, and performs monocular depth prediction on the original monocular picture to obtain a foreground depth map; the conversion module 20 generates a mask depth map and a background depth map according to the foreground depth map obtained by the prediction module 10, and performs mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map; the distortion module 30 obtains the distortion change values ​​corresponding to the left and right eye views, respectively, and deforms the original monocular picture according to the distortion change value and the target depth map obtained by the conversion module to obtain a binocular picture corresponding to the original monocular picture.

[0070] like Figure 3 As shown, this embodiment also provides an electronic device, including a memory 301 and a processor 302, wherein the memory 301 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 302 to implement the above-mentioned method of generating a binocular image based on a monocular image.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0072] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement the above-mentioned method of generating a binocular image based on a monocular image when executed.

[0073] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 301 and executed by the processor 302, and the input interface 305 and the output interface 306 complete the I / O interface transmission of data to complete the present invention. The one or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0074] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a memory 301 and a processor 302. Those skilled in the art may understand that this embodiment is only an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include an input device 307, a network access device, a bus, etc.

[0075] The processor 302 may be a central processing unit (CPU), or other general-purpose processors 302, digital signal processors 302 (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 302 may be a microprocessor 302, or the processor 302 may be any conventional processor 302, etc.

[0076] The memory 301 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory 301 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the memory 301 may include both an internal storage unit of the computer device and an external storage device. The memory 301 is used to store computer programs and other programs and data required by the computer device. The memory 301 may also be used to temporarily store in the output device 308. The aforementioned storage media include various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory ROM 303, a random access memory RAM 304, a disk or an optical disk.

[0077] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for generating a binocular image based on a monocular image, It is characterized in that The following steps are involved: Acquire an original monocular picture, and perform monocular depth prediction on the original monocular picture to obtain a foreground depth map; Generate a mask depth map and a background depth map according to the foreground depth map, and perform mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map; The distortion change values ​​corresponding to the left and right eye views are obtained respectively, and the original monocular image is deformed according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image.

2. The method for generating a binocular image based on a monocular image according to claim 1, It is characterized in that Generating a mask depth map according to the foreground depth map includes: Processing the foreground depth map using an edge detection algorithm to obtain an edge detection image, and determining whether the edge detection image is generated normally; When the edge detection image is generated normally, converting the edge detection image into a binary image using a color threshold algorithm, and determining whether the binary image is generated normally; When the binary image is generated normally, smoothing the binary image using a fuzzy algorithm to obtain a smoothed binary image, and determining whether the smoothed binary image is generated normally; When the smoothed binary image is generated normally, a color threshold algorithm is used to perform background extraction on the smoothed binary image to obtain a mask depth map.

3. The method for generating a binocular image based on a monocular image according to claim 1, It is characterized in that Generating a background depth map according to the foreground depth map includes: The foreground depth map is smoothed using a fuzzy algorithm to obtain a background depth map.

4. The method for generating a binocular image based on a monocular image according to claim 1, It is characterized in that The step of performing mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map includes: The foreground depth map, the mask depth map and the background depth map are sampled respectively, and all the sampling results are mixed and calculated according to the mask value to output a target depth map.

5. The method for generating a binocular image based on a monocular image according to claim 1, It is characterized in that The step of obtaining the distortion change values ​​corresponding to the left and right eye views respectively, and deforming the original monocular image according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image, includes: Sampling the target depth map to obtain a first sampling result, and sampling the original monocular image according to a first channel value of the first sampling result to obtain a second sampling result; Calculate the distortion change values ​​corresponding to the left and right eye views respectively according to the focal length and interocular distance of the camera and the depth information of the second sampling result; Performing matrix transformation on the original monocular image according to the distortion change values ​​corresponding to the left and right eye views, respectively, to obtain left and right eye views; The transparency in the left and right eye views is extracted respectively, and the transparency is optimized using a transparency optimization algorithm to obtain a binocular image.

6. The method for generating a binocular image based on a monocular image according to claim 5, It is characterized in that The first channel value is depth information.

7. The method for generating a binocular image based on a monocular image according to claim 5, It is characterized in that The step of performing matrix transformation on the original monocular image according to the distortion change values ​​respectively corresponding to the left and right eye views to obtain the left and right eye views comprises: Performing matrix transformation on the original monocular image according to the distortion change values ​​corresponding to the left and right eye views, respectively, to obtain two sets of new pixel positions, and determining whether the two sets of new pixel positions are both beyond the image range; When the two sets of new pixel positions are within the image range, calculating the offset between the second sampling result and the two sets of new pixel positions; Creating a grid including all pixel coordinates in the second sampling result, and adding the offsets to the corresponding coordinates in the grid to obtain corresponding new coordinates of the left and right views; The second sampling results are remapped according to the new coordinates to obtain left-eye and right-eye views.

8. A device for generating a binocular image based on a monocular image, It is characterized in that include: A prediction module, used for acquiring an original monocular picture, and performing monocular depth prediction on the original monocular picture to obtain a foreground depth map; A conversion module, used to generate a mask depth map and a background depth map according to the foreground depth map, and perform mixed calculation on the foreground depth map, the mask depth map and the background depth map to obtain a target depth map; The distortion module is used to obtain the distortion change values ​​corresponding to the left and right eye views respectively, and deform the original monocular image according to the distortion change value and the target depth map to obtain a binocular image corresponding to the original monocular image.

9. An electronic device, It is characterized in that The invention comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for generating a binocular image based on a monocular image as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, It is characterized in that The computer program enables a computer to implement a method for generating a binocular image based on a monocular image as claimed in any one of claims 1 to 7 when executed.

Images