Image perspective method and device, electronic equipment, wearable equipment and storage medium

Abstract

According to the image perspective method and device, the electronic equipment, the wearable equipment and the storage medium provided by the invention, multiple images shot by the wearable equipment are acquired, and the multiple images comprise the black-and-white image and at least one color image; the black-and-white image and the at least one color image are images shot at the same time; respectively converting the at least one color image and the black-and-white image into a first color image and a first black-and-white image; the first color image and the first black-and-white image are a color image and a black-and-white image corresponding to the corresponding virtual viewpoint; for a virtual viewpoint, a perspective image corresponding to the virtual viewpoint is determined according to the brightness of the first color image corresponding to the virtual viewpoint and the brightness of the first black-and-white image, and the black-and-white image is acquired through the wearable device by using the characteristic of high resolution of the black-and-white image. The brightness information of the black-and-white image and the brightness information of the color image are fused, so that the resolution of the perspective image is improved, and the perspective effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image perspective method, device, electronic device, wearable device and storage medium. Background Art

[0002] Mixed reality technology (MR) is a further development of virtual reality technology. It can introduce information about real scenes into the virtual environment to enhance the realism of the user experience.

[0003] By setting a binocular camera on the wearable device, taking images through the binocular camera and projecting them onto the screen, the user can see the images projected on the screen to achieve visual perspective. In the related art, two color cameras are usually used as the input of the perspective source when achieving visual perspective, but the perspective effect is poor and cannot meet the actual needs of the human eye. Summary of the invention

[0004] The present invention provides an image perspective method, device, electronic device, wearable device and storage medium, which are used to solve the problem that the existing visual perspective has a poor perspective effect and cannot meet the actual needs of the human eye.

[0005] In a first aspect, the present invention provides an image perspective method, which is applied to a wearable device, and the method comprises:

[0006] Acquire a plurality of images captured by the wearable device, wherein the plurality of images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images captured at the same time;

[0007] Converting the at least one color image and the black-and-white image into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint;

[0008] For a virtual viewpoint, a perspective image corresponding to the virtual viewpoint is determined according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint.

[0009] Optionally, converting the at least one color image and the black-and-white image into a first color image and a first black-and-white image, respectively, includes:

[0010] For any color image, determine first depth information corresponding to each pixel in the color image, and convert the color image into the first color image according to the first depth information;

[0011] For the black-and-white image, second depth information corresponding to each pixel in the black-and-white image is determined, and the black-and-white image is converted into the first black-and-white image according to the second depth information.

[0012] Optionally, determining the perspective image corresponding to the virtual viewpoint according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint includes:

[0013] Determine a pixel correspondence between the first color image and the first black-and-white image through optical flow;

[0014] For any first pixel point in the first color image, determine a second pixel point in the first black-and-white image corresponding to the first pixel point according to the pixel correspondence relationship;

[0015] Determine a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel, and determine a fused brightness value of the first pixel according to the first brightness value and the second brightness value;

[0016] The perspective image corresponding to the virtual viewpoint is determined according to the fused brightness values ​​of the first pixels.

[0017] Optionally, determining a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel includes:

[0018] Determining a pyramid window size corresponding to the first pixel point according to high- and low-frequency feature information of the first pixel point; wherein the high- and low-frequency feature information includes high frequency or low frequency, and the pyramid window size corresponding to the high frequency is smaller than the pyramid window size corresponding to the low frequency;

[0019] A first brightness value corresponding to the first pixel point and a second brightness value corresponding to the second pixel point are determined according to the pyramid window size corresponding to the first pixel point.

[0020] Optionally, determining a fused brightness value of the first pixel according to the first brightness value and the second brightness value includes:

[0021] For any first pixel, weight values ​​corresponding to the first brightness value and the second brightness value are determined according to first occlusion information corresponding to the first pixel and second occlusion information corresponding to the second pixel; the first occlusion information indicates an occlusion condition of the first pixel on the first color image; the second occlusion information indicates an occlusion condition of the second pixel on the first black-and-white image;

[0022] A fused brightness value of the first pixel is determined according to the weight value, the first brightness value, and the second brightness value.

[0023] Optionally, the wearable device includes a black and white image imaging device and a color camera; there are two color cameras, corresponding to a first eye and a second eye respectively; the black and white image imaging device is arranged between the two color cameras; the color image includes a color image taken by the color camera corresponding to the first eye, and a color image taken by the color camera corresponding to the second eye; the virtual viewpoint includes a first eye viewpoint and a second eye viewpoint; the first color image includes a first sub-color image under the first eye viewpoint and a second sub-color image under the second eye viewpoint; the first black and white image includes a first sub-black and white image under the first eye viewpoint and a second sub-black and white image under the second eye viewpoint;

[0024] Determining first depth information corresponding to each pixel in the color image includes:

[0025] Determine first depth information corresponding to each pixel in the color image according to the color image and the black-and-white image;

[0026] Determining second depth information corresponding to each pixel in the black-and-white image includes:

[0027] When determining the first sub-black-and-white image under the viewpoint of the first eye, determining second depth information corresponding to each pixel in the black-and-white image according to the black-and-white image and the color image taken by the color camera corresponding to the first eye;

[0028] And / or, when determining the second sub-black-and-white image under the viewpoint of the second eye, the second depth information corresponding to each pixel in the black-and-white image is determined according to the black-and-white image and the color image taken by the color camera corresponding to the second eye.

[0029] Optionally, the wearable device includes a black-and-white image imaging device and a color camera; there is one color camera, and the color camera corresponds to the first eye; the black-and-white image imaging device corresponds to the second eye; the color image is a color image taken by the color camera corresponding to the first eye; the black-and-white image is a black-and-white image output by the black-and-white image imaging device corresponding to the second eye; the first color image includes a first sub-color image under the viewpoint of the first eye and a second sub-color image under the viewpoint of the second eye; the first black-and-white image includes a first sub-black-and-white image under the viewpoint of the first eye and a second sub-black-and-white image under the viewpoint of the second eye; the first pixel point is a pixel point in the second sub-black-and-white image under the viewpoint of the second eye, and the second pixel point is a pixel point in the second sub-color image under the viewpoint of the second eye;

[0030] Determining the perspective image corresponding to the virtual viewpoint according to the fused brightness values ​​of the first pixels includes:

[0031] For any first pixel in the second black-and-white sub-image under the second eye's viewpoint, extract color information of a second pixel corresponding to the first pixel from the second color sub-image under the second eye's viewpoint, and determine the extracted color information as the color information of the first pixel;

[0032] The perspective image corresponding to the second eye viewpoint is determined according to the fused brightness values ​​of the first pixels and the color information.

[0033] In a second aspect, the present invention provides an image perspective device, which is applied to a wearable device, and includes:

[0034] An acquisition module, used to acquire a plurality of images captured by the wearable device, wherein the plurality of images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images captured at the same time;

[0035] A conversion module, used to convert the at least one color image and black-and-white image into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint;

[0036] The determination module is used to determine, for a virtual viewpoint, a perspective image corresponding to the virtual viewpoint according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint.

[0037] In a third aspect, the present invention provides an electronic device, comprising: at least one processor and a memory;

[0038] The memory stores computer-executable instructions;

[0039] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method as described in any one of the first aspects.

[0040] In a fourth aspect, the present invention provides a wearable device comprising a computing unit, a black and white image imaging device and at least one color camera; the black and white image imaging device is a depth sensor and / or a black and white camera; the computing unit is used to execute the method described in any one of the first aspects.

[0041] In a fifth aspect, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, a method as described in any one of the first aspects is implemented.

[0042] In a sixth aspect, the present invention provides a computer program product, comprising a computer program, which implements any method described in the first aspect when executed by a processor.

[0043] The present invention provides an image perspective method, device, electronic device, wearable device and storage medium. The method obtains multiple images taken by a wearable device, wherein the multiple images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images taken simultaneously; the at least one color image and the black-and-white image are respectively converted into a first color image and a first black-and-white image; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint; for a virtual viewpoint, the perspective image corresponding to the virtual viewpoint is determined according to the brightness of the first color image corresponding to the virtual viewpoint and the brightness of the first black-and-white image. The method utilizes the high resolution of black-and-white images, obtains black-and-white images through a wearable device, and fuses the brightness information of the black-and-white images with the brightness information of the color images to improve the resolution of the perspective images and improve the perspective effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] Figure 1 An application scenario diagram of an image perspective method provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a flow chart of an image perspective method provided by an embodiment of the present invention;

[0047] Figure 3 A schematic flow chart of another image perspective method provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of an imaging device with two color cameras and one black-and-white image provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a color camera and a black-and-white imaging device provided in an embodiment of the present invention;

[0050] Figure 6 A schematic structural diagram of an image perspective device provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

[0052] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0053] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0054] VR (Virtual Reality) refers to the technology that uses computer graphics systems and various reality and control interface devices to provide an immersive feeling in a three-dimensional interactive environment generated by a computer. Usually, a completely closed virtual space is formed through wearable devices, such as head-mounted displays.

[0055] Some VR products provide a see-through function, which allows users to interact with the outside world without removing the head-mounted display, thereby introducing information about real scenes into the virtual environment to enhance the realism of the user experience.

[0056] In order to realize the function of visual perspective of VR products, it is necessary to set an imaging device and a screen on the VR product, obtain images through the imaging device, and project the images onto the screen for users to watch. In the prior art, the imaging device set on the wearable device is two color cameras, and the images taken by the two color cameras are projected onto two screens respectively for users to watch. However, when two color cameras are used as the input of the perspective source, since the Bayer filter is set in the color camera, part of the light will be reflected when passing through the color camera, while the Bayer filter is not set in the black and white image imaging device, and the light entering the black and white image imaging device will not be reflected. Therefore, the signal-to-noise ratio of the color image is lower than that of the black and white image. In addition, the resolution of the color image is lower than that of the black and white image. Therefore, when two color images are used as the input source of perspective, the perspective effect will be poor, which cannot meet the needs of the human eye.

[0057] Based on the above problems, this application considers taking advantage of the fact that black and white images have richer details, higher resolution and higher signal-to-noise ratio, and fusing the brightness information of black and white images with the brightness information of color images to improve the resolution of the final perspective image. In a dark scene, there is less light in the scene. If a color camera is used, there will be less light in the color image formed. Therefore, this method can especially improve the overall effect of the perspective image in a dark scene.

[0058] Figure 1 An application scenario diagram of an image perspective method provided by an embodiment of the present invention, such as Figure 1 As shown, a black and white image imaging device, a color camera (at least one) and a computing unit are provided on the wearable device. The computing unit can generate a first color image and a first black and white image from the viewpoint of the human eye according to the captured image, wherein the left eye and the right eye correspond to a group of first color images and a first black and white image respectively, and then the two images are pixel aligned and the brightness information is fused to generate a perspective image, and displayed on the corresponding screen.

[0059] Figure 2 The present invention provides a flowchart of an image perspective method according to an embodiment of the present invention. The method is applied to an image perspective device, and the image perspective device is arranged on an electronic device. The method includes steps S201 to S203:

[0060] Step S201: Acquire multiple images taken by the wearable device, where the multiple images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images taken at the same time.

[0061] An imaging device is provided on the wearable device, and the imaging device can be used to capture images for viewing by the user. The wearable device can be a head-mounted display, and the imaging device can be provided outside the head-mounted display. In order to improve the effect of the perspective image, the imaging device provided here can include a black and white image imaging device, and at least one color camera, so that a black and white image and at least one color image can be obtained.

[0062] Optionally, the black and white image imaging device can be a black and white camera or a depth sensor, such as an iToF (indirect Time-of-Flight) sensor and a dToF (direct Time-of-Flight) sensor. The depth sensor can output a black and white image based on infrared imaging while outputting depth information. The black and white image based on infrared imaging has a higher signal-to-noise ratio and can effectively improve the resolution of the perspective image.

[0063] Optionally, at least one color camera and a black-and-white imaging device shoot images simultaneously to obtain at least one color image and one black-and-white image, thereby presenting a perspective image at any moment to the user.

[0064] For example, when a user wears the wearable device, the black and white image imaging device and at least one color camera can capture images in real time, or capture images simultaneously at preset intervals. Specifically, the time interval for capturing images can be adjusted according to the computing power of the computing unit in the wearable device.

[0065] For example, at a first moment, the black-and-white image imaging device and at least one color camera simultaneously capture images, and present the perspective image of the first moment to the user based on the simultaneously captured images; and, at a second moment 5 seconds after the first moment, the black-and-white image imaging device and at least one color camera simultaneously capture images, and present the perspective image of the second moment to the user based on the simultaneously captured images.

[0066] Step S202: convert the at least one color image and black-and-white image into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint.

[0067] Since the imaging device is arranged on the outside of the wearable device, and the wearable device needs to be placed on the user's head, the position of the imaging device is different from the position of the human eye, and the image taken by the camera is different from the image actually viewed by the human eye. Therefore, the image obtained by the imaging device cannot be directly presented to the human eye.

[0068] In order to allow the human eye to see an accurate image, it is necessary to convert at least one color image and a black-and-white image, that is, convert the color image into a first color image under a virtual viewpoint, and convert the black-and-white image into a first black-and-white image under a virtual viewpoint. The virtual viewpoint here refers to the human eye. Since a person has two eyes, the virtual viewpoint includes the first human eye viewpoint and the second human eye viewpoint. Therefore, it is necessary to obtain the first color image and the first black-and-white image corresponding to the first human eye viewpoint and the second human eye viewpoint, respectively.

[0069] Step S203: for a virtual viewpoint, determine a perspective image corresponding to the virtual viewpoint according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint.

[0070] After acquiring the first color image and the first black-and-white image at the same virtual viewpoint, the brightness of the first color image can be fused with the brightness of the first black-and-white image. In order to meet the actual needs of the human eye, the brightness of the first black-and-white image can be fused with the brightness of the first color image based on the first color image, and the fused brightness can be used as the brightness information of the first color image, and the first color image fused with the brightness of the first black-and-white image can be projected onto a corresponding screen to obtain a perspective image.

[0071] For example, when the brightness of two images is fused, a neural network fusion method, a Gaussian pyramid fusion method, an α-blending fusion method, and the like may be used.

[0072] Among them, two screens are set in the wearable device, corresponding to the first human eye viewpoint and the second human eye viewpoint respectively. When the first color image fused with the brightness of the first black and white image is projected, it is also necessary to project it to the corresponding screen. For example, the first color image corresponding to the left eye can be projected to the left screen, and the first color image corresponding to the right eye can be projected to the right screen.

[0073] The present invention provides an image perspective method, which obtains multiple images taken by a wearable device, wherein the multiple images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images taken at the same time; the at least one color image and the black-and-white image are converted into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint; for a virtual viewpoint, the perspective image corresponding to the virtual viewpoint is determined according to the brightness of the first color image corresponding to the virtual viewpoint and the brightness of the first black-and-white image. The method utilizes the high resolution of black-and-white images, sets a black-and-white image imaging device in the wearable device, and fuses the brightness information of the black-and-white image with the brightness information of the color image to improve the resolution of the perspective image and improve the perspective effect.

[0074] Figure 3 A schematic flow chart of another image perspective method provided by an embodiment of the present invention, the method comprising steps S301 to S303:

[0075] Step S301: Acquire multiple images taken by the wearable device, where the multiple images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images taken at the same time.

[0076] The implementation process of step S301 can refer to step S201 and will not be repeated here.

[0077] Step S302: for any color image, determine first depth information corresponding to each pixel in the color image, and convert the color image into the first color image according to the first depth information.

[0078] When determining a first color image from a human eye perspective based on a color image captured by a color camera, it is necessary to first calculate first depth information of each pixel in the color image to calculate the first color image based on the first depth information.

[0079] Optionally, when determining the depth information of any image, each imaging device needs to be calibrated first. The following takes the imaging device as a camera as an example for explanation. Each camera has a horizontal direction and a vertical direction. Through calibration, the horizontal directions of each camera can be parallel to avoid the problem of inaccurate calculated depth information caused by errors in the industrial installation process. Among them, when calibrating each camera, it can be calibrated according to the parameters calibrated during the installation process of each camera.

[0080] When calculating the depth information, it can be determined based on the two images. Since the two images are taken by cameras at different positions, the two images are slightly different. The first depth information can be determined based on the pixel value of each pixel in the two images.

[0081] The process of calculating depth information may be referred to as stereo matching or binocular depth estimation. For example, when calculating the first depth information of a color image, a disparity map corresponding to the color image may be calculated based on the pixel information of the two images. The disparity map is composed of the disparity of each pixel point. The disparity is the difference in pixel values ​​between the corresponding points in the two images at a point in the three-dimensional scene.

[0082] Parallax is the difference in direction when two cameras observe the same target. The farther away from the camera, the smaller the parallax, and the closer to the camera, the larger the parallax. Therefore, after determining the parallax of any pixel, the depth information of the pixel can be further determined based on the parallax, and then the color image can be converted into the first color image from the human eye's perspective based on the depth information.

[0083] When calculating the first color image, it is necessary to calculate based on the positional relationship between the human eye and the camera, and the first depth information of each pixel point.

[0084] There are two ways to set up an imaging device in a wearable device. The first way is to set up two color cameras and a black-and-white imaging device. The setting positions of the two color cameras correspond to the two eyes respectively, and the black-and-white imaging device can be set between the two color cameras.

[0085] Optionally, the wearable device includes a black and white image imaging device and a color camera; there are two color cameras, corresponding to a first eye and a second eye respectively; the black and white image imaging device is arranged between the two color cameras; the color image includes a color image taken by the color camera corresponding to the first eye, and a color image taken by the color camera corresponding to the second eye; the virtual viewpoint includes a first eye viewpoint and a second eye viewpoint; the first color image includes a first sub-color image under the first eye viewpoint and a second sub-color image under the second eye viewpoint; the first black and white image includes a first sub-black and white image under the first eye viewpoint and a second sub-black and white image under the second eye viewpoint;

[0086] Determining first depth information corresponding to each pixel in the color image includes:

[0087] First depth information corresponding to each pixel in the color image is determined according to the color image and the black-and-white image.

[0088] Figure 4 A schematic diagram of an embodiment of the present invention providing two color cameras and a black and white image imaging device, such as Figure 4 As shown, the wearable device can generate color image 1, black and white image and color image 2. Since the images taken by cameras at close distances are relatively close, the error in calculating depth information is small. Therefore, a first sub-color image at a first eye viewpoint based on color image 1 can be obtained from color image 1 and the black and white image, and a second sub-color image at a second eye viewpoint based on color image 2 can be obtained from color image 2 and the black and white image.

[0089] By respectively calculating the images of the two color images at the virtual viewpoint based on the black and white image, the accuracy of the image at the determined virtual viewpoint can be improved.

[0090] Step S303: for the black-and-white image, determine second depth information corresponding to each pixel in the black-and-white image, and convert the black-and-white image into the first black-and-white image according to the second depth information.

[0091] Optionally, determining the second depth information corresponding to each pixel in the black-and-white image includes:

[0092] When determining the first sub-black-and-white image under the viewpoint of the first eye, determining second depth information corresponding to each pixel in the black-and-white image according to the black-and-white image and the color image taken by the color camera corresponding to the first eye;

[0093] And / or, when determining the second sub-black-and-white image under the viewpoint of the second eye, the second depth information corresponding to each pixel in the black-and-white image is determined according to the black-and-white image and the color image taken by the color camera corresponding to the second eye.

[0094] When there are two color cameras, the black-and-white image imaging device is disposed between the two color cameras. In this case, the following method may be used to determine the second depth information of each pixel in the black-and-white image.

[0095] like Figure 4 As shown, for a black-and-white image, the disparity corresponding to the black-and-white image can be determined based on the black-and-white image and the color image 1. Figure 1 , and then determine the second depth information 1 of each pixel in the black-and-white image, and then calculate the first sub-black-and-white image corresponding to the black-and-white image according to the second depth information 1; and calculate the disparity corresponding to the black-and-white image according to the black-and-white image and the color image 2 Figure 2 , and then determine the second depth information 2 of each pixel in the black-and-white image, and then calculate the second sub-black-and-white image corresponding to the black-and-white image according to the second depth information 2.

[0096] When calculating the second black-and-white image at two virtual viewpoints corresponding to the black-and-white image, it can be determined based on the two color images respectively to improve the accuracy of the first sub-black-and-white image and the second sub-black-and-white image at the determined virtual viewpoints.

[0097] There is also a second way to set the camera in the wearable device, which is to set a color camera and a black and white image imaging device, and the two cameras correspond to the first eye and the second eye respectively.

[0098] Optionally, the wearable device includes a black and white image imaging device and a color camera; there is one color camera, and the color camera corresponds to the first eye; the black and white image imaging device corresponds to the second eye; the color image is a color image taken by the color camera corresponding to the first eye; the black and white image is a black and white image output by the black and white image imaging device corresponding to the second eye; the first color image includes a first sub-color image from the viewpoint of the first eye and a second sub-color image from the viewpoint of the second eye; the first black and white image includes a first sub-black and white image from the viewpoint of the first eye and a second sub-black and white image from the viewpoint of the second eye; the first pixel point is a pixel point in the second sub-black and white image from the viewpoint of the second eye, and the second pixel point is a pixel point in the second sub-color image from the viewpoint of the second eye.

[0099] Figure 5 A schematic diagram of a color camera and a black-and-white imaging device provided by an embodiment of the present invention. Figure 5As shown, a first sub-color image corresponding to the first eye can be calculated based on the color image and the black-and-white image; and, a first sub-black-and-white image corresponding to the first eye can be calculated based on the color image and the black-and-white image; and, a second sub-color image corresponding to the second eye can be calculated based on the color image and the black-and-white image, and a second sub-black-and-white image corresponding to the second eye can be calculated based on the color image and the black-and-white image.

[0100] There are advantages and disadvantages for the above two camera settings, and the specific camera setting method can be selected according to the actual situation. When two color cameras and a black-and-white image imaging device are set, the electronic device can obtain color images corresponding to the two eyes respectively. In the subsequent algorithm, the processing method for each color image is the same, and the brightness information of the black-and-white image is used to supplement the brightness information of the color image, which is relatively simple in algorithm design. For the solution with one color camera and one black-and-white image imaging device, one side corresponds to the black-and-white image, and the color information needs to be filled in for the black-and-white image. There are slight differences in algorithm design when processing the images generated by the two cameras.

[0101] In addition, the bandwidth required varies depending on the number of cameras. For example, when two color cameras and one black-and-white image imaging device are used, if the bandwidth required for one color camera is 3 megabytes and the bandwidth required for one black-and-white image imaging device is 1 megabyte, the bandwidth required for this solution is 7 megabytes. When one black-and-white image imaging device and one color camera are used, the required bandwidth is 4 megabytes. It can be seen that the solution of setting up two color cameras and one black-and-white image imaging device has higher bandwidth requirements for the hardware equipment. In addition, when the number of cameras set is large, the cost is also large. In practice, a specific camera setting method can be selected according to the actual situation.

[0102] After the above operation, no matter the scheme of two color cameras and one black-and-white imaging device or the scheme of one color camera and one black-and-white imaging device, two images under two virtual viewpoints can be obtained, namely, the first sub-color image and the first sub-black-and-white image, and the second sub-color image and the second sub-black-and-white image. In the following, the two images corresponding to one virtual viewpoint are collectively referred to as the first color image and the first black-and-white image.

[0103] Since the brightness information of the first black-and-white image needs to be used to enhance the brightness information of the first color image, the first color image and the first black-and-white image need to be pixel-aligned first so that the brightness of corresponding pixels can be fused.

[0104] Step S304: for a virtual viewpoint, determine the pixel correspondence between the first color image and the first black-and-white image through optical flow.

[0105] For the first color image and the first black-and-white image, since they are both images from the same virtual viewpoint, the two images are already in a roughly aligned state. However, after the roughly aligned state, there are still some differences between the two images. For example, for a certain position point in the real world, there will be some misalignment of the pixels in the first color image and the first black-and-white image, such as one or half a pixel. Therefore, it is necessary to finely align the first color image and the first black-and-white image.

[0106] Optionally, for two images, optical flow may be used to determine the distance that any pixel in one image moves in the other image, so that the first color image and the first black-and-white image may be precisely aligned according to the moving distance.

[0107] Since the resolution of the first black-and-white image is relatively high, during fine alignment, the first color image can be accurately aligned on the first black-and-white image.

[0108] Specifically, after precise alignment, the correspondence between each pixel in the first color image and each pixel in the first black and white image can be obtained. For example, the first pixel in the first color image is aligned with the second pixel in the first black and white image, and so on.

[0109] Step S305: for any first pixel in the first color image, determine a second pixel in the first black-and-white image corresponding to the first pixel according to the pixel correspondence relationship.

[0110] After the pixel correspondence is determined, the second pixel corresponding to any first pixel in the first color image in the first black-and-white image may be further determined according to the pixel correspondence, and then brightness fusion of the two corresponding pixels may be performed.

[0111] Step S306: determine a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel, and determine a fused brightness value of the first pixel according to the first brightness value and the second brightness value.

[0112] After determining the second pixel point in the first black and white image corresponding to the first pixel point in the first color image, the second brightness value can be fused with the first brightness value to obtain a fused brightness value, and the fused brightness value is determined as the brightness value of the first pixel point in the first color image.

[0113] Exemplarily, the first color image can be transformed into an HSV color space or a YUV color space. When transformed into an HSV color space, the second brightness value of the first black and white image is used as a V value to be merged with the V value (first brightness value) in the first color image; when the first color image is transformed into a YUV color space, the second brightness value of the first black and white image is used as a Y value to be merged with the Y value (first brightness value) in the first color image.

[0114] Optionally, determining a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel includes:

[0115] Determining a pyramid window size corresponding to the first pixel point according to high- and low-frequency feature information of the first pixel point; wherein the high- and low-frequency feature information includes high frequency or low frequency, and the pyramid window size corresponding to the high frequency is smaller than the pyramid window size corresponding to the low frequency;

[0116] A first brightness value corresponding to the first pixel point and a second brightness value corresponding to the second pixel point are determined according to the pyramid window size corresponding to the first pixel point.

[0117] When the brightness values ​​corresponding to two pixels are fused, a Gaussian pyramid fusion method can be used. When performing image fusion, the first color image and the first black-and-white image can be decomposed into a multi-scale pyramid image sequence using a Gaussian kernel to obtain N layers of images, and two images of the same layer are fused to obtain N fused images, and then the N fused images are reconstructed according to the inverse process of the pyramid image sequence generation to obtain a fused image.

[0118] When fusing two images of the same layer, there is a pyramid window size, and the brightness value of any pixel is related to the brightness values ​​of multiple pixels corresponding to the pyramid window size. The pyramid fusion method can use different pyramid windows at the low and high frequencies of the image, making the fused image smoother.

[0119] When performing pyramid fusion, when the pyramid window is set larger, the fusion result will have ghosting; when the pyramid window is set smaller, the fusion result will be truncated. Based on the above problems, it is necessary to set a suitable pyramid window size. Specifically, the high and low frequency feature information of the first pixel point can be determined. When it is high frequency, a smaller pyramid window size is set, and when it is low frequency, a larger pyramid window size is set.

[0120] When determining the high- and low-frequency feature information, Laplace transform may be performed on the first color image to obtain the high- and low-frequency feature information of any first pixel point.

[0121] After determining the pyramid window size corresponding to the first pixel, the first brightness value of the first pixel can be determined. The first brightness value is the average brightness of multiple pixels contained in a preset area in the first color image; the center of the preset area is the first pixel; the size of the preset area is the size of the pyramid window. Exemplarily, when the pyramid window size is determined to be 3*3, the average brightness of each pixel in the preset area of ​​3*3 with the first pixel in the first color image as the center can be used as the first brightness value of the first pixel. Similarly, the above method can be used to determine the second brightness value corresponding to the second pixel.

[0122] By determining the pyramid window size according to the high and low frequency feature information of the pixel point, different pyramid window sizes can be set at different pixel points, so as to accurately obtain the brightness values ​​corresponding to the first pixel point and the second pixel point, respectively, thereby improving the smoothness of the fused image.

[0123] Optionally, determining a fused brightness value of the first pixel according to the first brightness value and the second brightness value includes:

[0124] For any first pixel, weight values ​​corresponding to the first brightness value and the second brightness value are determined according to first occlusion information corresponding to the first pixel and second occlusion information corresponding to the second pixel; the first occlusion information indicates an occlusion condition of the first pixel on the first color image; the second occlusion information indicates an occlusion condition of the second pixel on the first black-and-white image;

[0125] A fused brightness value of the first pixel is determined according to the weight value, the first brightness value, and the second brightness value.

[0126] Since the first color image is obtained by processing a color image taken by a color camera, and the first black-and-white image is obtained by processing a black-and-white image acquired by a black-and-white image imaging device, and since the position of the camera is different from that of the human eye, there will be some occlusions in the processed first color image (the pixel information of the occlusions cannot be determined), and there will also be some occlusions in the first black-and-white image. When fusion is performed, the weight values ​​corresponding to the first brightness value and the second brightness value during fusion can be set according to the occlusion information of the first pixel point and the occlusion information of the second pixel point corresponding to the first pixel point.

[0127] Optionally, when the first occlusion information corresponding to the first pixel is occluded, and the second occlusion information corresponding to the second pixel is unoccluded, the weight value 1 corresponding to the first brightness value can be set to be smaller than the weight value 2 corresponding to the second brightness value, and the sum of the weight value 1 and the weight value 2 is 1.

[0128] For example, when the first occlusion information corresponding to the first pixel is occluded, and the second occlusion information corresponding to the second pixel is unoccluded, the weight value 1 corresponding to the first brightness value can be set to 0.3, and the weight value 2 corresponding to the second brightness value can be set to 0.7. The fused brightness value of the first pixel is obtained by weighted summation, that is, the product of the weight value 1 and the first brightness value is calculated, and the product of the weight value 2 and the second brightness value is calculated. The two products are added to obtain the fused brightness value.

[0129] Optionally, when the occlusion information corresponding to the first pixel and the second pixel are both unoccluded, the first gradient at the first pixel and the second gradient at the second pixel are calculated, and whether there is a detail feature at the position point is determined based on the magnitude relationship between the first gradient and the second gradient. When there is a detail feature, the weight value corresponding to the second brightness value is set greater than the weight value corresponding to the first brightness value; when there is no detail feature, the weight value corresponding to the first brightness value is set greater than the weight value corresponding to the second brightness value.

[0130] When determining whether there is a detail feature at the position point, the first gradient and the second gradient can be compared. When the second gradient is greater than the first gradient and the difference exceeds a preset value, it means that there is a detail feature at the position point. At this time, the high resolution of the details in the black and white image can be used to improve the resolution of the color image. Therefore, the weight value 2 corresponding to the second brightness value is set to be greater than the weight value 1 corresponding to the first brightness value. On the contrary, it means that the position point is relatively flat and there is no detail feature. At this time, the weight value corresponding to the first brightness value can be set to be greater than the weight value corresponding to the second brightness value.

[0131] Optionally, when the first occlusion information corresponding to the first pixel is occluded, and the second occlusion information corresponding to the second pixel is occluded, Gaussian filtering and smoothing may be used to complete the pixel information of the occluded portion.

[0132] By setting the weight value during fusion according to the occlusion conditions of the first color image and the first black-and-white image, the accuracy of the fused brightness value can be improved.

[0133] Step S307: determining a perspective image corresponding to the virtual viewpoint according to the fused brightness values ​​of the first pixels.

[0134] Since the RGB image is an image corresponding to the hardware, when the fused brightness value of each first pixel is determined, the HSV image or the YUV image can be obtained, and the HSV image or the YUV image can be further converted into an RGB image.

[0135] By first performing pixel alignment on the first color image and the first black-and-white image and then performing brightness fusion, the accuracy of the fusion result can be improved.

[0136] In addition, for the solution of setting up a color camera and a black and white image imaging device, an additional processing step is required when calculating the perspective image.

[0137] Optionally, determining the perspective image corresponding to the virtual viewpoint according to the fused brightness values ​​of the first pixels includes:

[0138] For any first pixel in the second black-and-white sub-image under the second eye's viewpoint, extract color information of a second pixel corresponding to the first pixel from the second color sub-image under the second eye's viewpoint, and determine the extracted color information as the color information of the first pixel;

[0139] The perspective image corresponding to the second eye viewpoint is determined according to the fused brightness values ​​of the first pixels and the color information.

[0140] For the three-camera solution, or for the color camera side in the two-camera solution, when brightness fusion is performed, the brightness of the first black-and-white image is fused to the first color image. For the two-camera solution, a black-and-white image imaging device is provided at the second eye. In this case, when brightness fusion is performed, the brightness of the second sub-color image can be fused to the second sub-black-and-white image. In addition, since there is no color information in the second sub-black-and-white image, the color information can be extracted from the second sub-color image, so that the perspective image under the viewpoint of the second eye can be obtained according to the extracted color information and the fused brightness value.

[0141] In the two-camera solution, when determining the perspective image from the second eye's viewpoint, by using the second black-and-white sub-image as a reference image, the second black-and-white sub-image is closer to the real image from the second eye's viewpoint, and then by extracting color information, the perspective image from the second eye's viewpoint can be determined more accurately.

[0142] Figure 6 A schematic diagram of the structure of an image perspective device provided by an embodiment of the present invention, wherein the device is applied to a wearable device, and the device 60 includes:

[0143] An acquisition module 601 is used to acquire a plurality of images captured by the wearable device, wherein the plurality of images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images captured at the same time;

[0144] A conversion module 602 is used to convert the at least one color image and black-and-white image into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint;

[0145] The determination module 603 is used to determine, for a virtual viewpoint, a perspective image corresponding to the virtual viewpoint according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint.

[0146] Optionally, the conversion module 602 is specifically used for:

[0147] For any color image, determine first depth information corresponding to each pixel in the color image, and convert the color image into the first color image according to the first depth information;

[0148] For the black-and-white image, second depth information corresponding to each pixel in the black-and-white image is determined, and the black-and-white image is converted into the first black-and-white image according to the second depth information.

[0149] Optionally, the determining module 603 is specifically configured to:

[0150] Determine a pixel correspondence between the first color image and the first black-and-white image through optical flow;

[0151] For any first pixel point in the first color image, determine a second pixel point in the first black-and-white image corresponding to the first pixel point according to the pixel correspondence relationship;

[0152] Determine a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel, and determine a fused brightness value of the first pixel according to the first brightness value and the second brightness value;

[0153] The perspective image corresponding to the virtual viewpoint is determined according to the fused brightness values ​​of the first pixels.

[0154] Optionally, when determining the first brightness value corresponding to the first pixel and the second brightness value corresponding to the second pixel, the determining module 603 is specifically configured to:

[0155] Determining a pyramid window size corresponding to the first pixel point according to high- and low-frequency feature information of the first pixel point; wherein the high- and low-frequency feature information includes high frequency or low frequency, and the pyramid window size corresponding to the high frequency is smaller than the pyramid window size corresponding to the low frequency;

[0156] A first brightness value corresponding to the first pixel point and a second brightness value corresponding to the second pixel point are determined according to the pyramid window size corresponding to the first pixel point.

[0157] Optionally, when determining the fused brightness value of the first pixel point according to the first brightness value and the second brightness value, the determining module 603 is specifically configured to:

[0158] For any first pixel, weight values ​​corresponding to the first brightness value and the second brightness value are determined according to first occlusion information corresponding to the first pixel and second occlusion information corresponding to the second pixel; the first occlusion information indicates an occlusion condition of the first pixel on the first color image; the second occlusion information indicates an occlusion condition of the second pixel on the first black-and-white image;

[0159] A fused brightness value of the first pixel is determined according to the weight value, the first brightness value, and the second brightness value.

[0160] Optionally, the wearable device includes a black and white image imaging device and a color camera; there are two color cameras, corresponding to a first eye and a second eye respectively; the black and white image imaging device is arranged between the two color cameras; the color image includes a color image taken by the color camera corresponding to the first eye, and a color image taken by the color camera corresponding to the second eye; the virtual viewpoint includes a first eye viewpoint and a second eye viewpoint; the first color image includes a first sub-color image under the first eye viewpoint and a second sub-color image under the second eye viewpoint; the first black and white image includes a first sub-black and white image under the first eye viewpoint and a second sub-black and white image under the second eye viewpoint;

[0161] When determining the first depth information corresponding to each pixel in the color image, the conversion module 602 is specifically used to:

[0162] Determine first depth information corresponding to each pixel in the color image according to the color image and the black-and-white image;

[0163] When determining the second depth information corresponding to each pixel in the black-and-white image, the conversion module 602 is specifically configured to:

[0164] When determining the first sub-black-and-white image under the viewpoint of the first eye, determining second depth information corresponding to each pixel in the black-and-white image according to the black-and-white image and the color image taken by the color camera corresponding to the first eye;

[0165] And / or, when determining the second sub-black-and-white image under the viewpoint of the second eye, the second depth information corresponding to each pixel in the black-and-white image is determined according to the black-and-white image and the color image taken by the color camera corresponding to the second eye.

[0166] Optionally, the wearable device includes a black-and-white image imaging device and a color camera; there is one color camera, and the color camera corresponds to the first eye; the black-and-white image imaging device corresponds to the second eye; the color image is a color image taken by the color camera corresponding to the first eye; the black-and-white image is a black-and-white image output by the black-and-white image imaging device corresponding to the second eye; the first color image includes a first sub-color image under the viewpoint of the first eye and a second sub-color image under the viewpoint of the second eye; the first black-and-white image includes a first sub-black-and-white image under the viewpoint of the first eye and a second sub-black-and-white image under the viewpoint of the second eye; the first pixel point is a pixel point in the second sub-black-and-white image under the viewpoint of the second eye, and the second pixel point is a pixel point in the second sub-color image under the viewpoint of the second eye;

[0167] When determining the perspective image corresponding to the virtual viewpoint according to the fused brightness values ​​of each first pixel point, the determining module 603 is specifically used to:

[0168] For any first pixel in the second black-and-white sub-image under the second eye's viewpoint, extract color information of a second pixel corresponding to the first pixel from the second color sub-image under the second eye's viewpoint, and determine the extracted color information as the color information of the first pixel;

[0169] The perspective image corresponding to the second eye viewpoint is determined according to the fused brightness values ​​of the first pixels and the color information.

[0170] The image perspective device provided by the embodiment of the present invention can realize the above-mentioned Figure 2 and Figure 3 The image perspective method of the embodiment shown has similar implementation principles and technical effects, which will not be described in detail here.

[0171] Figure 7 The hardware structure diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the electronic device provided by this embodiment includes: at least one processor 701 and a memory 702. The processor 701 and the memory 702 are connected via a bus 703.

[0172] In a specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the method in the above method embodiment.

[0173] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0174] In the above Figure 7In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0175] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0176] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0177] The present invention also provides a wearable device, comprising a computing unit, a black and white image imaging device and at least one color camera; the black and white image imaging device comprises a depth sensor and / or a black and white camera, and the computing unit is used to execute the method of the above method embodiment.

[0178] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of the above method embodiment is implemented.

[0179] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of the above method embodiment when executed by a processor.

[0180] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0181] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0182] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0183] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0184] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0185] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image perspective method, characterized in that: The method is applied to a wearable device, and the method comprises: Acquire a plurality of images captured by the wearable device, wherein the plurality of images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images captured at the same time; Converting the at least one color image and the black-and-white image into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint; For a virtual viewpoint, a perspective image corresponding to the virtual viewpoint is determined according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint.

2. The method according to claim 1, characterized in that Converting the at least one color image and the black-and-white image into a first color image and a first black-and-white image, respectively, comprises: For any color image, determine first depth information corresponding to each pixel in the color image, and convert the color image into the first color image according to the first depth information; For the black-and-white image, second depth information corresponding to each pixel in the black-and-white image is determined, and the black-and-white image is converted into the first black-and-white image according to the second depth information.

3. The method according to claim 1, characterized in that Determining a perspective image corresponding to the virtual viewpoint according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint includes: Determine a pixel correspondence between the first color image and the first black-and-white image through optical flow; For any first pixel point in the first color image, determine a second pixel point in the first black-and-white image corresponding to the first pixel point according to the pixel correspondence relationship; Determine a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel, and determine a fused brightness value of the first pixel according to the first brightness value and the second brightness value; The perspective image corresponding to the virtual viewpoint is determined according to the fused brightness values ​​of the first pixels.

4. The method according to claim 3, characterized in that Determining a first brightness value corresponding to the first pixel and a second brightness value corresponding to the second pixel includes: Determining a pyramid window size corresponding to the first pixel point according to high- and low-frequency feature information of the first pixel point; wherein the high- and low-frequency feature information includes high frequency or low frequency, and the pyramid window size corresponding to the high frequency is smaller than the pyramid window size corresponding to the low frequency; A first brightness value corresponding to the first pixel point and a second brightness value corresponding to the second pixel point are determined according to the pyramid window size corresponding to the first pixel point.

5. The method according to claim 3, characterized in that: Determining a fused brightness value of the first pixel according to the first brightness value and the second brightness value includes: For any first pixel, weight values ​​corresponding to the first brightness value and the second brightness value are determined according to first occlusion information corresponding to the first pixel and second occlusion information corresponding to the second pixel; the first occlusion information indicates an occlusion condition of the first pixel on the first color image; the second occlusion information indicates an occlusion condition of the second pixel on the first black-and-white image; A fused brightness value of the first pixel is determined according to the weight value, the first brightness value, and the second brightness value.

6. The method according to any one of claims 2 to 5, characterized in that: The wearable device includes a black and white image imaging device and a color camera; there are two color cameras, corresponding to a first eye and a second eye respectively; the black and white image imaging device is arranged between the two color cameras; the color image includes a color image taken by the color camera corresponding to the first eye, and a color image taken by the color camera corresponding to the second eye; the virtual viewpoint includes a first eye viewpoint and a second eye viewpoint; the first color image includes a first sub-color image under the first eye viewpoint and a second sub-color image under the second eye viewpoint; the first black and white image includes a first sub-black and white image under the first eye viewpoint and a second sub-black and white image under the second eye viewpoint; Determining first depth information corresponding to each pixel in the color image includes: Determine first depth information corresponding to each pixel in the color image according to the color image and the black-and-white image; Determining second depth information corresponding to each pixel in the black-and-white image includes: When determining the first sub-black-and-white image under the viewpoint of the first eye, determining second depth information corresponding to each pixel in the black-and-white image according to the black-and-white image and the color image taken by the color camera corresponding to the first eye; And / or, when determining the second sub-black-and-white image under the viewpoint of the second eye, the second depth information corresponding to each pixel in the black-and-white image is determined according to the black-and-white image and the color image taken by the color camera corresponding to the second eye.

7. The method according to claim 3, characterized in that The wearable device includes a black-and-white image imaging device and a color camera; there is one color camera, and the color camera corresponds to a first eye; the black-and-white image imaging device corresponds to a second eye; the color image is a color image taken by the color camera corresponding to the first eye; the black-and-white image is a black-and-white image output by the black-and-white image imaging device corresponding to the second eye; the first color image includes a first sub-color image under the viewpoint of the first eye and a second sub-color image under the viewpoint of the second eye; the first black-and-white image includes a first sub-black-and-white image under the viewpoint of the first eye and a second sub-black-and-white image under the viewpoint of the second eye; the first pixel point is a pixel point in the second sub-black-and-white image under the viewpoint of the second eye, and the second pixel point is a pixel point in the second sub-color image under the viewpoint of the second eye; Determining the perspective image corresponding to the virtual viewpoint according to the fused brightness values ​​of the first pixels includes: For any first pixel in the second black-and-white sub-image under the second eye's viewpoint, extract color information of a second pixel corresponding to the first pixel from the second color sub-image under the second eye's viewpoint, and determine the extracted color information as the color information of the first pixel; The perspective image corresponding to the second eye viewpoint is determined according to the fused brightness values ​​of the first pixels and the color information.

8. An image perspective device, characterized in that: The device is applied to a wearable device, and comprises: An acquisition module, used to acquire a plurality of images captured by the wearable device, wherein the plurality of images include a black-and-white image and at least one color image; the black-and-white image and the at least one color image are images captured at the same time; A conversion module, used to convert the at least one color image and black-and-white image into a first color image and a first black-and-white image respectively; the first color image and the first black-and-white image are the color image and the black-and-white image corresponding to the corresponding virtual viewpoint; The determination module is used to determine, for a virtual viewpoint, a perspective image corresponding to the virtual viewpoint according to the brightness of the first color image and the brightness of the first black-and-white image corresponding to the virtual viewpoint.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 7.

10. A wearable device, characterized in that: It comprises a computing unit, a black and white image forming device and at least one color camera; the black and white image forming device is a depth sensor and / or a black and white camera, and the computing unit is used to execute the method as claimed in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images