Head-mounted devices, image processing methods, apparatuses, storage media, and software products

Abstract

This disclosure relates to a head-mounted device, image processing method, apparatus, storage medium, and program product, which can enhance the display format of the head-mounted device's perspective function and improve the user experience. The head-mounted device includes: two single-channel cameras for capturing two single-channel images; a multi-channel camera for capturing color images; a processor for acquiring depth information of the external scene based on the two single-channel images and fusing the depth information with the color images to obtain a color image with depth information; and a display screen for displaying the color image with depth information.

Description

Technical Field

[0001] This disclosure relates to the field of wearable technology, and more particularly to a head-mounted device, an image processing method, an apparatus, a storage medium, and a program product. Background Technology

[0002] With the development of wearable technology, most head-mounted devices now have see-through capabilities. Specifically, head-mounted devices can acquire images of the real-world scene through a front-facing camera and display these images on a screen, allowing users to see the real-world scene while wearing the device.

[0003] However, due to the current structural design of head-mounted devices, when the head-mounted device is in see-through mode, it displays black and white images, which is too simplistic and reduces the user's sense of realism. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a head-mounted device, an image processing method, an apparatus, a storage medium, and a program product.

[0005] A first aspect of this disclosure provides a head-mounted device, comprising: two single-channel cameras for capturing two single-channel images; a multi-channel camera for capturing color images; a processor for acquiring depth information of an external scene based on the two single-channel images, and fusing the depth information with the color images to obtain a color image with depth information; and a display screen for displaying the color image with depth information.

[0006] Optionally, the multi-channel camera is positioned on the vertical line connecting the two single-channel cameras.

[0007] Optionally, the two single-channel cameras are positioned at the positions corresponding to the user's two eyes when wearing the headset.

[0008] Optionally, the overlapping area of ​​the fields of view of the two cameras is greater than or equal to the area threshold.

[0009] Optionally, the head-mounted device further includes: a fill light assembly disposed around the multi-channel camera; the fill light assembly is used to provide supplementary lighting for the multi-channel camera when the ambient light intensity of the head-mounted device is less than or equal to an intensity threshold.

[0010] A second aspect of this disclosure provides an image processing method, comprising: capturing two single-channel images using two single-channel cameras; capturing a color image using a multi-channel camera; obtaining depth information of an external scene based on the two single-channel images, and fusing the depth information with the color image to obtain a color image with depth information; and displaying the color image with depth information.

[0011] Optionally, the method further includes: when the ambient light intensity is less than or equal to an intensity threshold, determining a supplementary lighting scheme for the supplementary lighting assembly based on the ambient light intensity, the supplementary lighting assembly being disposed around the multi-channel camera; and controlling at least one supplementary light in the supplementary lighting assembly to emit light according to the supplementary lighting scheme.

[0012] Optionally, before determining the supplementary lighting scheme of the supplementary lighting assembly based on the ambient light intensity when the ambient light intensity is less than or equal to the intensity threshold, the method further includes: determining the ambient light intensity based on the exposure parameters of the multi-channel camera or the average grayscale value of the color image captured by the multi-channel camera.

[0013] A third aspect of this disclosure provides an image processing apparatus, comprising: an image capturing module, an image processing module, and a display module; the image capturing module is configured to capture two single-channel images using two single-channel cameras and capture a color image using a multi-channel camera; the image processing module is configured to obtain depth information of an external scene based on the two single-channel images captured by the image capturing module, and fuse the depth information with the color image captured by the image capturing module to obtain a color image with depth information; the display module is configured to display the color image with depth information obtained by the image processing module.

[0014] Optionally, the supplementary lighting assembly and the image processing device further include: a determining module and a controlling module; the determining module is used to determine a supplementary lighting scheme for the supplementary lighting assembly based on the ambient light intensity when the ambient light intensity is less than or equal to an intensity threshold, the supplementary lighting assembly being disposed around the multi-channel camera; the controlling module is used to control at least one supplementary lighting lamp in the supplementary lighting assembly to emit light according to the supplementary lighting scheme determined by the determining module.

[0015] Optionally, the determining module is further configured to determine the ambient light intensity based on the exposure parameters of the multi-channel camera or the average grayscale value of the color image captured by the multi-channel camera before determining the supplementary lighting scheme of the supplementary lighting assembly based on the ambient light intensity when the ambient light intensity is less than or equal to the intensity threshold.

[0016] A fourth aspect of this disclosure provides a head-mounted device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image processing method as described in the first aspect.

[0017] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method as described in the first aspect.

[0018] A sixth aspect of this disclosure provides a computer program product, wherein the computer program product includes a computer program that, when the computer program product is run on a processor, causes the processor to execute the computer program to implement the image processing method as described in the first aspect.

[0019] A seventh aspect of this disclosure provides a chip including a processor and a communication interface coupled to the processor, the processor being configured to execute program instructions to implement the image processing method as described in the first aspect.

[0020] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: In this disclosure, depth information of the external scene can be obtained from two single-channel images captured by two single-channel cameras, and this depth information is fused with a color image captured by a multi-channel camera to obtain a color image with depth information; and this color image with depth information is then displayed. Thus, when the head-mounted device is in perspective mode, the user can see a stereoscopic color image of the external scene, increasing the display format of the head-mounted device's perspective function and improving the user experience. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of a planar layout of multiple cameras in a head-mounted device provided in an embodiment of this disclosure;

[0024] Figure 2This is a schematic diagram of another planar layout of multiple cameras in a head-mounted device provided in an embodiment of this disclosure;

[0025] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this disclosure;

[0026] Figure 4 This is a structural block diagram of an image processing apparatus provided in an embodiment of the present disclosure;

[0027] Figure 5 This is a structural block diagram of a head-mounted device provided in an embodiment of the present disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] The terms "first," "second," etc., used in this disclosure and in the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The head-mounted device in this embodiment can be a head-mounted device with VR, AR, or MR functions, such as a VR headset, VR glasses, VR helmet, AR glasses, AR helmet, MR glasses, MR helmet, etc. The specific device can be determined according to the actual situation, and this embodiment does not limit it.

[0032] This disclosure provides a head-mounted device, which includes: two single-channel cameras for capturing two single-channel images; a multi-channel camera for capturing color images; a processor for acquiring depth information of an external scene based on the two single-channel images, and fusing the depth information with the color images to obtain a color image with depth information; and a display screen for displaying the color image with depth information.

[0033] The multi-channel camera can be a camera for capturing RGB images, a camera for capturing YUV images, a camera for capturing YCbCr images, a camera for capturing HSI images, or a camera for capturing HSV images, etc. The specific camera can be determined according to actual usage requirements, and this disclosure does not limit the specific camera.

[0034] It is understood that the two single-channel cameras are positioned at different locations on the head-mounted device, allowing them to capture images of the external scene from different angles. Therefore, the processor can obtain depth information of the external scene based on the images captured by the two single-channel cameras. Then, based on this depth information and the color images captured by the multi-channel cameras, a color image with depth information is obtained, i.e., a three-dimensional color image. The display screen displays this color image with depth information, allowing the user to view it. Thus, when the head-mounted device is in perspective mode, the user can view a color image with depth information, increasing the display format of the head-mounted device's perspective function and improving the user experience.

[0035] The display based on the color image with depth information is as follows: the head-mounted device acquires two 3D color images based on the human eye's display principle using the color image with depth information, and then displays these two 3D color images. In other words, the head-mounted device uses a virtual camera to capture two 3D color images based on the color image with depth information, and then displays these two 3D color images through two display channels. The user's two eyes are aligned with the corresponding two display channels to view the 3D color image displayed on those channels. These two 3D color images share a common area, which is the area shared by both eyes when viewing the color image with depth information. These two 3D color images can be combined to form the color image with depth information.

[0036] Optionally, each of the two single-channel cameras can be an existing part of the head-mounted device's design structure, or it can be added to the existing design structure of the head-mounted device. The specific design structure can be determined according to the actual situation, and this disclosure does not limit it.

[0037] In this embodiment of the disclosure, when both single-channel cameras are already present in the design structure of the head-mounted device, this solution only adds a multi-channel camera to the original design, which enables the head-mounted device to display a stereoscopic color image of the external scene when in the perspective function mode. Moreover, the hardware and software improvements to the head-mounted device are minimal, which can reduce improvement costs, improve improvement efficiency, and does not affect the implementation of other functions of the head-mounted device.

[0038] Optionally, in addition to the two single-channel cameras, the head-mounted device may also include other single-channel cameras, which can be determined according to actual needs, and this disclosure does not limit the specific cameras.

[0039] Optionally, the multi-channel camera can be positioned between the two single-channel cameras, on one side of the two single-channel cameras, or in other locations. The specific location can be determined according to actual needs, and this disclosure does not limit the specific location.

[0040] Optionally, the multi-channel camera is positioned on the vertical line connecting the two single-channel cameras.

[0041] For example, the multi-channel camera can be set at the midpoint of the line connecting the two single-channel cameras, or it can be set above the vertical line connecting the two single-channel cameras, or it can be set below the vertical line connecting the two single-channel cameras. The specific location can be determined according to the actual situation, and this embodiment does not limit it.

[0042] In this embodiment, the multi-channel camera is positioned on the vertical line connecting the two single-channel cameras. This facilitates the fusion of the depth information and the color image, reducing image processing (such as anti-distortion processing) required to prevent image distortion. The process is simple and improves the efficiency of acquiring color images with depth information. Furthermore, with the multi-channel camera positioned on the vertical line connecting the two single-channel cameras, there are no display blind spots when displaying the color image with depth information. Compared to existing methods, this eliminates display blind spots on the screen, improving the user experience.

[0043] Optionally, the two single-channel cameras are positioned at the positions corresponding to the user's two eyes when wearing the headset.

[0044] The position of the user's two eyes when wearing the head-mounted device can be the position directly facing the user's eyes when wearing the head-mounted device, or it can be a position at a distance less than or equal to a distance threshold from the position directly facing the user's eyes when wearing the head-mounted device. The specific position can be determined according to the actual situation, and this embodiment does not limit it.

[0045] When a user wears the head-mounted device, the projection of the position on the head-mounted device directly facing the eyes onto the plane of the user's face falls on the position of the user's eyes. The position directly facing the two eyes can be the position of the eyeballs, the position of the area of ​​the eyes, or other feasible situations, which can be determined according to the actual situation, and this disclosed embodiment does not limit it.

[0046] The distance threshold can be set according to the actual situation, and this embodiment does not limit it.

[0047] In this embodiment of the disclosure, the two single-channel cameras are respectively positioned at the positions corresponding to the user's two eyes when wearing the head-mounted device. This makes the final obtained color image with depth information closer to the image content obtained when the human eye directly views the external scene, thereby improving the user experience.

[0048] Furthermore, by placing the two single-channel cameras at positions corresponding to the user's eyes when wearing the headset, the depth information of various objects in the external scene obtained from the two single-channel images captured by the two single-channel cameras can correspond to the two-dimensional information of various objects in the color image captured by the multi-channel camera. The resulting fused color image with depth information is more accurate, which can reduce dizziness for the user and improve the user experience.

[0049] Optionally, the overlapping area of ​​the fields of view of the two cameras is greater than or equal to the area threshold.

[0050] The regional threshold can be determined according to the actual situation, and this disclosure does not limit it.

[0051] The overlapping area of ​​the fields of view of the two cameras can also be called the common field of view of the two cameras.

[0052] Since inaccurate depth information in an image can cause distortion of the displayed scene, in this embodiment of the present disclosure, appropriately increasing the overlapping area of ​​the fields of view of the two cameras can improve the accuracy of the depth information of the image, thereby ensuring that the scene is not distorted.

[0053] Optionally, in this embodiment of the disclosure, the common field of view of the two single-channel cameras can be increased by appropriately adjusting the yaw angle of the two single-channel cameras.

[0054] Optionally, the head-mounted device further includes: a fill light assembly disposed around the multi-channel camera; the fill light assembly is used to provide supplementary lighting for the multi-channel camera when the ambient light intensity of the head-mounted device is less than or equal to an intensity threshold.

[0055] The supplementary lighting assembly may include one or more supplementary lights. Each of the one or more supplementary lights may be set in different directions of the multi-channel camera, or the one or more supplementary lights may be set in the same position of the multi-channel camera, or other settings may be used. The specific settings can be determined according to the actual situation, and this disclosure does not limit them.

[0056] Optionally, the one or more fill lights can be arranged in a ring around the multi-channel camera, or arranged in a line at one end of the multi-channel camera, or arranged in other ways around the multi-channel camera. The specific arrangement can be determined according to the actual situation, and this disclosure does not limit the specific arrangement.

[0057] It is understood that the distance between each supplementary light and the multi-channel camera is less than or equal to a preset threshold. The preset threshold can be determined according to the actual situation, and this embodiment does not limit it.

[0058] It is understandable that when the ambient light intensity of the head-mounted device is less than or equal to the intensity threshold, the fill light component can be turned on to provide supplementary lighting for the multi-channel camera. When the ambient light intensity is greater than the intensity threshold, there is no need to provide supplementary lighting for the multi-channel camera, and the fill light component can be turned off.

[0059] Optionally, when the ambient light intensity of the head-mounted device is less than or equal to the intensity threshold, and the ambient light intensity is different, at least one of the following can be used to provide appropriate supplementary lighting for the multi-channel camera: adjusting the brightness of the supplementary lights turned on in the supplementary light assembly, adjusting the number of supplementary lights turned on in the supplementary light assembly, and adjusting the color of the supplementary lights turned on in the supplementary light assembly.

[0060] Optionally, the ambient light intensity can be detected by a photosensitive sensor, or determined by the average grayscale value of a color image captured by a multi-channel camera, or determined by the exposure parameters when the multi-channel camera captures a color image. The specific determination can be made according to the actual situation, and this disclosure does not limit it.

[0061] In this embodiment, a fill light assembly is provided around the multi-channel camera to ensure that the multi-channel camera can capture clearer color images of the external scene in a darker environment. This allows users to see the surrounding environment clearly through the head-mounted device, thus improving the user experience.

[0062] For example, such as Figure 1The diagram shows a planar layout of multiple cameras in a head-mounted device. Circles represent single-channel cameras, blank squares represent multi-channel cameras, and black squares represent fill light assemblies. Markings "11" and "12" indicate two single-channel cameras used to capture two single-channel images in this embodiment (the other two single-channel cameras are cameras with other functions in the head-mounted device (e.g., positioning, handle recognition, etc.)). Marking "13" indicates a multi-channel camera used to capture color images in this embodiment, and marking "14" indicates a fill light assembly used to provide supplementary lighting for the multi-channel cameras in this embodiment.

[0063] For example, such as Figure 2 The diagram shows another planar layout of multiple cameras in a head-mounted device. Circles represent single-channel cameras, blank squares represent multi-channel cameras, and black squares represent supplementary lighting components. Markings "21" and "22" indicate two single-channel cameras used to capture two single-channel images in this embodiment. These two single-channel cameras are positioned at the position corresponding to the user's eyes when wearing the head-mounted device (the other two single-channel cameras are cameras with other functions in the head-mounted device (e.g., positioning functions)). Marking "23" indicates a multi-channel camera used to capture color images in this embodiment, and marking "24" indicates a supplementary lighting component used to provide supplementary lighting for the multi-channel cameras in this embodiment.

[0064] The execution subject of the image processing method provided in this embodiment can be the head-mounted device described above, or it can be a functional module and / or functional entity in the head-mounted device that can implement the image processing method. The specific implementation subject can be determined according to actual usage requirements, and this embodiment does not limit it.

[0065] like Figure 3 As shown in the figure, this disclosure provides an image processing method, which may include the following steps 301 to 305.

[0066] 301. Take two single-channel images using two single-channel cameras.

[0067] For example, two single-channel images are captured using two single-channel cameras on a head-mounted device.

[0068] 302. Capture color images using a multi-channel camera.

[0069] For example, color images are captured using a multi-channel camera on a head-mounted device.

[0070] 303. Obtain the depth information of the external scene based on the two single-channel images.

[0071] 304. Fuse the depth information with the color image to obtain a color image with depth information.

[0072] The depth of feature points can be calculated from the common viewing area in the two single-channel images. This depth information is then projected onto the color image to obtain a color image with depth information. Image distortion can be eliminated through processes such as distortion correction.

[0073] 305. Display based on the color image with depth information.

[0074] In this embodiment, depth information of the external scene can be obtained from two single-channel images captured by two single-channel cameras, and this depth information is fused with a color image captured by a multi-channel camera to obtain a color image with depth information; this color image with depth information is then displayed. Thus, when the head-mounted device is in perspective mode, the user can see a stereoscopic color image of the external scene, increasing the display options for the perspective function of the head-mounted device and improving the user experience.

[0075] Optionally, before step 302 above, the image processing method provided in this application embodiment may further include steps 306 to 307 as described below.

[0076] 306. When the ambient light intensity is less than or equal to the intensity threshold, determine the supplementary lighting scheme of the supplementary lighting assembly based on the ambient light intensity.

[0077] The fill light assembly is positioned around the multi-channel camera.

[0078] For example, when the ambient light intensity of the head-mounted device is less than or equal to an intensity threshold, the supplementary lighting scheme of the head-mounted device's supplementary lighting assembly is determined based on the ambient light intensity.

[0079] 307. According to the supplementary lighting scheme, control at least one supplementary light in the supplementary lighting assembly to emit light.

[0080] In this embodiment of the disclosure, when the ambient light intensity is less than or equal to the intensity threshold, at least one of the fill lights in the fill light assembly is controlled to emit light according to the ambient light intensity. This ensures that the images captured by the multi-channel camera are relatively clear and that the images captured are not unclear due to the surrounding environment being too dark, thereby affecting the user's view of the external environment and reducing the user's experience of using the perspective function.

[0081] Optionally, prior to step 306 above, the image processing method provided in this application embodiment may further include step 308 below.

[0082] 308. Determine the ambient light intensity based on the exposure parameters of the multi-channel camera or the average grayscale value of the color image captured by the multi-channel camera.

[0083] It's understandable that after capturing a color image using a multi-channel camera, the average grayscale value of the color image can be calculated. Then, based on the average grayscale value and the correspondence between the average grayscale value and ambient light intensity, the current ambient light intensity can be determined. Generally, the smaller the average grayscale value of the image, the lower the ambient light intensity.

[0084] It's understandable that after capturing a color image using a multi-channel camera, the exposure parameters of the image can be obtained. Then, based on these exposure parameters and the relationship between the image's exposure parameters and ambient light intensity, the current ambient light intensity can be determined. Generally, the higher the image's exposure parameters, the lower the ambient light intensity.

[0085] The system determines whether supplemental lighting is needed based on the ambient light intensity. If the ambient light intensity is greater than the intensity threshold, it determines that no supplemental lighting is needed, acquires the next frame image, and determines whether supplemental lighting is needed based on the next frame image. If the ambient light intensity is less than or equal to the intensity threshold, it determines the supplemental lighting scheme and controls the supplemental lighting component to provide supplemental lighting to the multi-channel camera according to the supplemental lighting scheme.

[0086] The supplementary lighting scheme includes at least one of the following: the number of supplementary lights to be turned on, the distribution of the supplementary lights to be turned on, the color of the supplementary lights to be turned on, and the intensity of the supplementary lights to be turned on. The specific supplementary lighting scheme can be determined according to the actual situation, and this embodiment does not limit it.

[0087] Figure 4 This is a structural block diagram of an image processing apparatus shown in an embodiment of the present disclosure, such as... Figure 4 As shown, the device includes: an image capturing module 401, an image processing module 402, and a display module 403; the image capturing module 401 is used to capture two single-channel images through two single-channel cameras and to capture a color image through a multi-channel camera; the image processing module 402 is used to obtain depth information of the external scene based on the two single-channel images captured by the image capturing module 401, and to fuse the depth information with the color image captured by the image capturing module 401 to obtain a color image with depth information; the display module 403 is used to display the color image with depth information obtained by the image processing module 402.

[0088] Optionally, the image processing device further includes: a determining module and a controlling module; the determining module is used to determine a supplementary lighting scheme for the supplementary lighting assembly based on the ambient light intensity when the ambient light intensity is less than or equal to an intensity threshold, the supplementary lighting assembly being disposed around the multi-channel camera; the controlling module is used to control at least one supplementary lighting lamp in the supplementary lighting assembly to emit light according to the supplementary lighting scheme determined by the determining module.

[0089] Optionally, the determining module is further configured to determine the ambient light intensity based on the exposure parameters of the multi-channel camera or the average grayscale value of the color image captured by the multi-channel camera before determining the supplementary lighting scheme of the supplementary lighting assembly based on the ambient light intensity when the ambient light intensity is less than or equal to the intensity threshold.

[0090] It should be noted that the image processing device in this embodiment can be the head-mounted device in the above embodiments, or it can be a functional module and / or functional entity in the head-mounted device in the above embodiments that can implement the image processing method provided in the above method embodiments. The specific device can be determined according to the actual situation and is not limited here.

[0091] In this embodiment, each module can implement the image processing method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0092] Figure 5 This is a schematic diagram of a head-mounted device provided in an embodiment of the present disclosure. It is used to illustrate any image processing method implemented in the embodiments of the present disclosure and should not be construed as a specific limitation on the embodiments of the present disclosure.

[0093] like Figure 5 As shown, the head-mounted device 500 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the head-mounted device 500. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0094] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows head-mounted device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although a head-mounted device 500 with various devices is shown, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0095] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it can perform the functions defined in any image processing method provided in embodiments of this disclosure.

[0096] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0097] In some implementations, the client and server can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0098] The aforementioned computer-readable medium may be included in the aforementioned head-mounted device; or it may exist independently and not assembled into the head-mounted device.

[0099] The aforementioned computer-readable medium carries one or more programs that, when executed by the head-mounted device, cause the head-mounted device to: capture two single-channel images using two single-channel cameras; capture a color image using a multi-channel camera; obtain depth information of the external scene based on the two single-channel images, and fuse the depth information with the color image to obtain a color image with depth information; and display the color image with depth information.

[0100] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0103] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0104] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0106] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0107] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A head-mounted device, characterized in that, The head-mounted device includes: Two single-channel cameras are used to capture two single-channel images. The two single-channel cameras are respectively set at the positions corresponding to the two eyes when the user wears the head-mounted device. The overlapping area of ​​the field of view of the two single-channel cameras is greater than or equal to the area threshold. A multi-channel camera for capturing color images, wherein the multi-channel camera is positioned on the vertical line of the line connecting the two single-channel cameras; The processor is configured to acquire depth information of the external scene based on the two single-channel images, and fuse the depth information with the color image to obtain a three-dimensional color image with depth information; The display screen is used to acquire two three-dimensional color images based on the human eye display principle of the three-dimensional color image with depth information, and to display the two three-dimensional color images. The two three-dimensional color images have a common area, which is the shared viewing area of ​​the two eyes when the human eye looks at the color image with depth information.

2. The head-mounted device according to claim 1, characterized in that, The head-mounted device also includes: a fill light assembly disposed around the multi-channel camera; The supplementary lighting assembly is used to provide supplementary lighting for the multi-channel camera when the ambient light intensity of the head-mounted device is less than or equal to an intensity threshold.

3. An image processing method, characterized in that, The method includes: Two single-channel images are captured by two single-channel cameras, which are respectively positioned at the positions corresponding to the user's two eyes when wearing the head-mounted device. The overlapping area of ​​the fields of view of the two single-channel cameras is greater than or equal to the area threshold. Color images are captured by a multi-channel camera, which is positioned on the vertical line connecting the two single-channel cameras. Depth information of the external scene is obtained from the two single-channel images, and the depth information is fused with the color image to obtain a three-dimensional color image with depth information; Two three-dimensional color images are obtained based on the principle of human eye display in three-dimensional color images with depth information, and the two three-dimensional color images are displayed. The two three-dimensional color images have a common area, which is the shared viewing area of ​​the two eyes when the human eye looks at the color image with depth information.

4. The method according to claim 3, characterized in that, The method further includes: When the ambient light intensity is less than or equal to the intensity threshold, the supplementary lighting scheme of the supplementary lighting assembly is determined according to the ambient light intensity, and the supplementary lighting assembly is arranged around the multi-channel camera; According to the aforementioned supplementary lighting scheme, at least one supplementary light in the supplementary lighting assembly is controlled to emit light.

5. The method according to claim 4, characterized in that, Before determining the supplementary lighting scheme for the supplementary lighting assembly based on the ambient light intensity when the ambient light intensity is less than or equal to an intensity threshold, the method further includes: The ambient light intensity is determined based on the exposure parameters of the multi-channel camera or the average grayscale value of the color image captured by the multi-channel camera.

6. An image processing apparatus, characterized in that, The device includes: an image capturing module, an image processing module, and a display module; The image capturing module is used to capture two single-channel images through two single-channel cameras and capture color images through a multi-channel camera. The two single-channel cameras are respectively set at the positions corresponding to the two eyes when the user wears the head-mounted device. The multi-channel camera is set on the vertical line of the line connecting the two single-channel cameras. The overlapping area of ​​the field of view of the two single-channel cameras is greater than or equal to the area threshold. The image processing module is used to obtain depth information of the external scene based on the two single-channel images captured by the image capturing module, and to fuse the depth information with the color image captured by the image capturing module to obtain a three-dimensional color image with depth information. The display module is used to acquire two three-dimensional color images based on the human eye display principle of the three-dimensional color image with depth information, and to display the two three-dimensional color images. The two three-dimensional color images have a common area, which is the shared viewing area of ​​the two eyes when the human eye looks at the color image with depth information.

7. A head-mounted device, characterized in that, include: Memory and processor; memory is used to store computer programs. The processor is used to execute the image processing method of any one of claims 4 to 5 when a computer program is invoked.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 4 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image processing method according to any one of claims 4 to 5.

Images