Image display method and device, head-mounted display equipment and medium

Abstract

The invention provides an image display method and device, head-mounted display equipment and a medium, and is applied to the head-mounted display equipment, and the head-mounted display equipment comprises a camera module and a distance measurement module. The method comprises the steps that distance information corresponding to all pixel points in a real world image collected by the camera module is obtained through measurement of the distance measurement module; comparing the distance information corresponding to each pixel point in the real world image with a virtual image distance of a corresponding pixel point in a to-be-displayed virtual image of the head-mounted display device to obtain a comparison result corresponding to each pixel point; and fusing the to-be-displayed virtual image and the real world image according to the comparison result corresponding to each pixel point to obtain a fused image, and displaying the fused image.

Description

Technical Field

[0001] This disclosure relates to the field of head-mounted display technology, and more specifically, to an image display method, an image display device, a head-mounted display device, and a computer-readable storage medium. Background Technology

[0002] Currently, head-mounted display devices such as Virtual Reality (VR) devices typically have a video perspective mode. Video perspective uses the camera module on the VR device to transmit images of the real world captured by the camera module to the VR device's display screen with extremely low latency that is almost imperceptible to the human eye. In this way, users can see images of the real world through the camera module.

[0003] However, since head-mounted displays cannot distinguish between the occlusion relationships between objects in the real world and objects in the virtual world, there are counterintuitive phenomena such as imbalance of proportions and "near objects appearing smaller and far objects appearing larger," which affect the viewing experience of video perspective and the immersive feeling of combining the virtual and real worlds. Summary of the Invention

[0004] One objective of this disclosure is to provide a new technical solution for image display.

[0005] According to a first aspect of the present disclosure, an image display method is provided, applied to a head-mounted display device, the head-mounted display device including a camera module and a ranging module, the method comprising:

[0006] The distance information corresponding to each pixel in the real-world image captured by the camera module is obtained by measuring the distance using the ranging module.

[0007] By comparing the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device, the comparison results corresponding to each pixel are obtained.

[0008] The virtual image to be displayed and the real-world image are merged based on the comparison results corresponding to each pixel to obtain a merged image and the merged image is then displayed.

[0009] Optionally, before obtaining the distance information corresponding to each pixel in the real-world image acquired by the camera module through the ranging module, the method further includes:

[0010] While the user is wearing the head-mounted display device, it is detected whether the head-mounted display device is in video perspective mode;

[0011] When the head-mounted display device is in video perspective mode, the ranging module is activated.

[0012] Optionally, the step of fusing the virtual image to be displayed and the real-world image based on the comparison results corresponding to each pixel to obtain a fused image and displaying the fused image includes:

[0013] Based on the comparison results between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed, the target display rules for any pixel in the real-world image and the corresponding pixel in the virtual image to be displayed are determined.

[0014] According to the target display rules, any pixel in the real-world image and the corresponding pixel in the virtual image to be displayed are fused to obtain a fused image;

[0015] The fused image is displayed.

[0016] Optionally, determining the target display rules for any pixel in the real-world image and the corresponding pixel in the virtual image to be displayed based on the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed includes:

[0017] The comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is that the distance information corresponding to any pixel in the real-world image is less than the virtual image distance of the corresponding pixel in the virtual image to be displayed. Therefore, the target display rule is determined to be to display any pixel in the real-world image.

[0018] The comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is that the distance information corresponding to any pixel in the real-world image is greater than the virtual image distance of the corresponding pixel in the virtual image to be displayed. Therefore, the target display rule is to display the corresponding pixel in the virtual image to be displayed.

[0019] The comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is that the distance information corresponding to any pixel in the real-world image is equal to the virtual image distance of the corresponding pixel in the virtual image to be displayed. Therefore, the target display rule is determined to be to display either any pixel in the real-world image or the corresponding pixel in the virtual image to be displayed.

[0020] Optionally, the head-mounted display device further includes an inertial measurement unit, and the method further includes:

[0021] Acquire the inertial data collected by the inertial measurement unit;

[0022] The head-mounted display device is located using the inertial data and the real-world image.

[0023] Optionally, after fusing the virtual image to be displayed and the real-world image according to the comparison results corresponding to each pixel, obtaining a fused image and displaying the fused image, the method further includes:

[0024] The distance information corresponding to each pixel in the new real-world image acquired by the camera module is obtained by measuring the distance using the ranging module.

[0025] Optionally, the ranging module is a depth camera, and the number of the depth cameras is greater than or equal to 1 and less than or equal to 4.

[0026] According to a second aspect of the present disclosure, an image display device is provided, applied to a head-mounted display device, the head-mounted display device including a camera module and a ranging module, the device comprising:

[0027] The measurement module is used to measure the distance information corresponding to each pixel in the real-world image captured by the camera module through the ranging module;

[0028] The comparison module is used to compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device, and obtain the comparison result corresponding to each pixel.

[0029] The fusion module is used to fuse the virtual image to be displayed and the real-world image according to the comparison results corresponding to each pixel, so as to obtain a fused image and display the fused image.

[0030] According to a third aspect of the present disclosure, a head-mounted display device is provided, the head-mounted display device including a camera module and a ranging module, the head-mounted display device further including:

[0031] Memory is used to store executable computer instructions;

[0032] A processor, configured to execute the image display method according to the first aspect above, under the control of the executable computer instructions.

[0033] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, perform the method described in the first aspect above.

[0034] One beneficial effect of this disclosure is that the head-mounted display device can acquire real-world images of the user's scene through its own camera module, measure the distance information corresponding to each pixel in the real-world image through its own ranging module, and compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device to obtain the comparison result corresponding to each pixel. Then, based on the comparison results corresponding to each pixel, the virtual image to be displayed and the real-world image are fused to obtain and display a fused image. In other words, the head-mounted display device can measure the distance information of objects existing in the real world during video perspective and can accurately determine the occlusion relationship between the virtual image and the real-world image, making the display effect of the head-mounted display device more realistic.

[0035] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0037] Figure 1 This is a schematic diagram of the hardware configuration of a head-mounted display device according to an embodiment of the present disclosure;

[0038] Figure 2 This is a schematic flowchart of an image display method according to an embodiment of the present disclosure;

[0039] Figure 3 This is a flowchart illustrating an example of an image display method according to this disclosure;

[0040] Figure 4 This is a schematic block diagram of an image display device according to an embodiment of the present disclosure;

[0041] Figure 5 This is a schematic block diagram of a head-mounted display device according to an embodiment of the present disclosure. Detailed Implementation

[0042] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the embodiments of the present disclosure.

[0043] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0044] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0045] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0046] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0047] <Hardware Configuration>

[0048] Figure 1 This is a block diagram of the hardware configuration of a head-mounted display device 1000 according to an embodiment of the present disclosure.

[0049] like Figure 1 As shown, the head-mounted display device 1000 typically has a video perspective function. The head-mounted display device 100 can be a VR device, an MR (Mixed Reality) device, etc., and this disclosure does not limit it.

[0050] In one embodiment, such as Figure 1 As shown, the head-mounted display device 1000 may include a processor 1100, a memory 1200, an interface module 1300, a communication module 1400, a display module 1500, a camera module 1600, a ranging module 1700, and a speaker 1800, etc. The processor 1100 may include, but is not limited to, a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), non-volatile memory such as a hard disk, etc. The interface module 1300 may include, for example, various bus interfaces, such as serial bus interfaces (including USB interfaces), parallel bus interfaces, etc. The communication module 1400 may be capable of wired or wireless communication, for example. The display module 1500 may be, for example, an LCD screen, an LED screen, a touch screen, etc. The camera module 1600 is used to acquire real-world images of the scene where the head-mounted display device is located. The ranging module 1700 is used to measure distance information of real-world objects in the scene where the head-mounted display device is located. The speaker 1800 can be used to output voice information.

[0051] In this embodiment, the memory 1200 of the head-mounted display device 1000 is used to store instructions for controlling the processor 1100 to operate in order to implement or support the implementation of an image display method according to any embodiment. Those skilled in the art can design instructions based on the schemes disclosed in this specification. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0052] Those skilled in the art should understand that, although in Figure 1 The present specification shows a number of devices for a head-mounted display device 1000. However, the head-mounted display device 1000 of the embodiments described herein may involve only some of the devices, or may include other devices, which is not limited herein.

[0053] Figure 1 The head-mounted display device 1000 shown is illustrative only and is not intended to limit this specification, its application, or its use.

[0054] Hereinafter, various embodiments and examples according to the present disclosure will be described with reference to the accompanying drawings.

[0055] <Method Implementation>

[0056] Figure 2 An image display method according to an embodiment of this disclosure is shown, applied to a head-mounted display device, which is typically a head-mounted display device with video see-through functionality. The image display method of this embodiment can be, for example, by... Figure 1 The illustrated head-mounted display device 1000 can be a VR device, and typically includes a camera module and a ranging module.

[0057] like Figure 2 As shown, the image display method provided in this embodiment may include the following steps S2100 to S2300.

[0058] Step S2100: The distance information corresponding to each pixel in the real-world image acquired by the camera module is obtained by measuring the distance using the ranging module.

[0059] The head-mounted display device includes a camera module for capturing real-world images of the environment in which the head-mounted display device is located. Furthermore, the camera module includes at least one six-degree-of-freedom (6DOF) camera; for example, the camera module may include four 6DOF cameras.

[0060] The head-mounted display device includes a ranging module, which measures the distance to real-world objects in the scene in which the head-mounted display device is located. Typically, the ranging module can be, but is not limited to, an ultrasonic testing module, a laser ranging module, an infrared ranging module, a pulse ranging module, or a radar ranging module. This embodiment does not limit this; however, those skilled in the art will understand that any module with ranging capabilities and capable of accurately inputting distance information to the head-mounted display device can serve as its ranging module.

[0061] In an optional embodiment, when the ranging module is a laser ranging module, the ranging module can be a depth camera, and the number of the depth cameras is greater than or equal to one and less than or equal to four. Typically, when the ranging module includes one or two depth cameras, the depth cameras can be positioned in the center area of ​​the head-mounted display device's surface, for example, in the exact center of the surface, or slightly above or below the center. When the ranging module includes three or four depth cameras, the depth cameras can be positioned at the corners of the head-mounted display device's surface.

[0062] The depth camera can be either an indirect time-of-flight (ITOF) camera or a direct time-of-flight (DTOF) camera; this embodiment is not limited to either. Generally, a depth camera calculates the distance to an object by measuring the time difference between the transmission and reception of a pulsed light signal. A depth camera typically includes a transmitter and a receiver. The transmitter emits a pulsed light signal, and the receiver receives the returned pulsed light signal. When the transmitter emits a pulsed light signal and illuminates an object, a portion of the pulsed light signal is reflected back to the receiver. By measuring the time difference between transmission and reception, the depth camera can calculate the distance to the object.

[0063] Typically, depth cameras are used to capture depth images of real-world objects within the scene where a head-mounted display (HMD) is located. The pixels in this depth image represent depth information, specifically the distances of real-world objects within the HMD's scene to the depth camera. Furthermore, the depth camera and the HMD's own camera system are usually calibrated at the factory to ensure that the depth image captured by the depth camera accurately reflects the distance information of each pixel in the real-world image captured by the camera system.

[0064] In an optional embodiment, before performing step S2100 to measure the distance information corresponding to each pixel in the real-world image acquired by the camera module through the ranging module, the image display method of this embodiment further includes: detecting whether the head-mounted display device is in video perspective mode when the user is wearing the head-mounted display device; and activating the ranging module when the head-mounted display device is in video perspective mode.

[0065] As described in the background section, video perspective utilizes the camera module on a VR device to transmit images of the real world captured by the camera module to the VR device's display screen with extremely low latency, almost imperceptible to the human eye. This allows the user to see images of the real world through the camera module. In other words, the image display method of this embodiment requires that the head-mounted display device be in video perspective mode.

[0066] Among them, head-mounted display devices with video perspective function can automatically enter video perspective mode after powering on. Of course, if the video perspective function of the head-mounted display device is turned off, the user can also trigger the video perspective function of the head-mounted display device to be turned on.

[0067] Specifically, when a user wears a head-mounted display, the display detects whether its video perspective function is enabled. If the video perspective function is enabled, the head-mounted display activates the ranging module. With the ranging module activated, it can measure the distance information corresponding to each pixel in the real-world image captured by the camera module.

[0068] After obtaining the distance information corresponding to each pixel in the real-world image captured by the camera module through the ranging module, the process proceeds to:

[0069] Step S2200: Compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device to obtain the comparison result corresponding to each pixel.

[0070] In this context, the virtual image to be displayed is an image of an object in the virtual world. The virtual image distance of any pixel in the virtual image can be understood as the equivalent distance from that pixel to the human eye. This equivalent distance represents the perceived distance of an object of the same size to the human eye in the real world. Typically, the virtual image distances corresponding to each pixel in the virtual image to be displayed are known values.

[0071] The comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed can include one of the following: the distance information corresponding to any pixel in the real-world image is less than the virtual image distance of the corresponding pixel in the virtual image to be displayed; the distance information corresponding to any pixel in the real-world image is greater than the virtual image distance of the corresponding pixel in the virtual image to be displayed; or the distance information corresponding to any pixel in the real-world image is equal to the virtual image distance of the corresponding pixel in the virtual image to be displayed.

[0072] Specifically, after obtaining the distance information corresponding to each pixel in the real-world image and the virtual image distance corresponding to each pixel in the virtual image to be displayed, the head-mounted display device can match each pixel in the real-world image with each pixel in the virtual image to be displayed. Then, it compares the distance information of each pixel in the real-world image with the virtual distance of the corresponding pixel in the virtual image to be displayed, and obtains the comparison result corresponding to each pixel.

[0073] In one example, the head-mounted display device can assign distance information of real-world objects in the scene where the head-mounted display device is located, measured by the ranging module, to corresponding pixels in the real-world image, so that each pixel carries the corresponding distance information. In this way, the head-mounted display device can directly compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed.

[0074] In another example, if the ranging module is a depth camera, the so-called comparison of the distance information corresponding to each pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed can also be understood as comparing the depth information corresponding to each pixel in the depth image corresponding to the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed.

[0075] After comparing the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device, and obtaining the comparison results for each pixel, the process proceeds to:

[0076] Step S2300: Based on the comparison results corresponding to each pixel, fuse the virtual image to be displayed and the real-world image to obtain a fused image and display the fused image.

[0077] In this embodiment, after obtaining the distance information corresponding to each pixel in the real-world image and the comparison result between the virtual image distance corresponding to the corresponding pixel in the virtual image to be displayed, the head-mounted display device can perform fusion processing on the virtual image to be displayed and the real-world image according to the comparison result corresponding to each pixel, obtain the fused image, and display the fused image through the display screen of the head-mounted display device.

[0078] In an optional embodiment, step S2300, which involves fusing the virtual image to be displayed and the real-world image based on the comparison results corresponding to each pixel, to obtain a fused image and display the fused image, may further include: determining a target display rule for any pixel in the real-world image and a corresponding pixel in the virtual image to be displayed based on the comparison results between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed; fusing any pixel in the real-world image and a corresponding pixel in the virtual image to be displayed based on the target display rule to obtain a fused image; and displaying the fused image.

[0079] Specifically, this step, based on the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed, determines the target display rule for the real-world image and the corresponding pixel in the virtual image to be displayed. This can further include: if the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is less than the virtual image distance of the corresponding pixel in the virtual image to be displayed, then the target display rule is determined to be to display any pixel in the real-world image.

[0080] If the comparison between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is greater than the virtual image distance of the corresponding pixel in the virtual image to be displayed, then the target display rule is determined to be to display the corresponding pixel in the virtual image to be displayed.

[0081] The comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is that the distance information corresponding to any pixel in the real-world image is equal to the virtual image distance of the corresponding pixel in the virtual image to be displayed. Therefore, the target display rule is determined to be to display either any pixel in the real-world image or the corresponding pixel in the virtual image to be displayed.

[0082] It should be noted that if the distance information corresponding to any pixel in the real-world image is less than the virtual image distance of the corresponding pixel in the virtual image to be displayed, it indicates that any pixel in the real-world image is closer to the human eye and should be displayed first.

[0083] If the distance information corresponding to any pixel in a real-world image is greater than the virtual image distance of the corresponding pixel in the virtual image to be displayed, it indicates that the corresponding pixel in the virtual image to be displayed is closer to the human eye and should be displayed first.

[0084] If the distance information corresponding to any pixel in the real-world image is equal to the virtual image distance of the corresponding pixel in the virtual image to be displayed, it indicates that the distance from any pixel in the real-world image and the corresponding pixel in the virtual image to be displayed is the same as the distance from the human eye. Either any pixel in the real-world image or the corresponding pixel in the virtual image to be displayed can be displayed according to actual needs.

[0085] According to embodiments of this disclosure, a head-mounted display device can acquire real-world images of the user's scene using its own camera module, measure the distance information corresponding to each pixel in the real-world image using its own ranging module, and compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device to obtain the comparison result for each pixel. Then, based on the comparison results for each pixel, the virtual image to be displayed and the real-world image are fused to obtain and display a fused image. In other words, the head-mounted display device can measure the distance information of objects existing in the real world during video perspective and can accurately determine the occlusion relationship between the virtual image and the real-world image, making the display effect of the head-mounted display device more realistic.

[0086] In one embodiment, the head-mounted display device further includes an inertial measurement unit (IMU) for acquiring inertial data of the head-mounted display device. Here, the image display method of this disclosure embodiment further includes: acquiring the inertial data acquired by the inertial measurement unit; and positioning the head-mounted display device using the inertial data and the real-world image.

[0087] Specifically, when a user is wearing a head-mounted display device and the head-mounted display device is in video perspective mode, the head-mounted display device can determine its pose by using inertial data collected by the inertial measurement unit and real-world images collected by the camera device, so as to achieve positioning and tracking of the head-mounted display device.

[0088] In one embodiment, after performing the above step S2300 to fuse the virtual image to be displayed and the real-world image according to the comparison results corresponding to each pixel, and to obtain and display the fused image, the image display method of this embodiment further includes: measuring the distance information corresponding to each pixel in the new real-world image acquired by the camera module through the ranging module.

[0089] Taking the depth camera as an example, after the head-mounted display displays the fused image, while the video perspective function of the head-mounted display continues to be used, the depth camera will continuously scan and detect the scene in which the head-mounted display is located to cover the complex scene where the surrounding environment is constantly changing. For example, if a new object enters the ranging range, it ensures that the head-mounted display's perception of the user's surrounding environment is always up-to-date when the user is wearing the head-mounted display.

[0090] <Example>

[0091] The following is an example of an image display method, see reference. Figure 3 The image display method includes:

[0092] Step 301: The user puts on the head-mounted display device, and the head-mounted display device starts working.

[0093] Step 302: The head-mounted display device detects whether it has entered video perspective mode, and if it detects that it has entered video perspective mode, it starts the ranging module.

[0094] Step 303: The ranging module of the head-mounted display device measures the distance information corresponding to each pixel in the real-world image captured by the camera module.

[0095] It should be noted that the head-mounted display device also uses its internal inertial measurement unit and camera device to calibrate and position itself.

[0096] Step 304: The head-mounted display device obtains the virtual image distance corresponding to each pixel in the virtual image to be displayed.

[0097] Step 305: The head-mounted display device compares and matches each pixel in the real-world image with each pixel in the virtual image to be displayed.

[0098] Step 306: The head-mounted display device determines the difference between the distance information corresponding to each pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed.

[0099] Step 307: The head-mounted display device determines that pixels closer to the human eye will be displayed first.

[0100] It should be noted that when a user uses the video perspective function, the image processing system inside the head-mounted display analyzes the difference between the distance between the real-world object being viewed and the user's eyes, and the distance between the virtual image and the user's eyes. If the distance between the virtual image and the user's eyes is less than the distance between the user's eyes and the real-world object, the image processing system does not display the real-world image captured by the camera; instead, it displays the virtual image within the virtual world. If the distance between the user's eyes and the real-world object is less than or equal to the distance between the virtual image and the user's eyes, the image processing system displays the real-world image captured by the camera, showing the actual object in the real world.

[0101] Step 308: The head-mounted display device obtains the display logic of the fused image accurate to the pixel level.

[0102] Step 309: The head-mounted display device merges the virtual image to be displayed and the real-world image, displays the merged image on the display screen, and continues to execute step 303.

[0103] Based on this example, it can measure the distance and locate objects in the real world during the perspective process by combining external scanning with logical judgment processing algorithms. At the same time, it can accurately determine the occlusion relationship between virtual images and real-world images, making the optical perspective display effect of head-mounted virtual reality products more realistic in scenarios such as virtual-real combination and real-world interaction.

[0104] <Device Embodiment>

[0105] Figure 4 This is a schematic diagram of the structure of an image display device according to one embodiment. Figure 4 As shown, the image display device 400 is applied to a head-mounted display device, which includes a camera module and a ranging module. The image display device 400 includes a measurement module 410, a comparison module 420, and a fusion module 430.

[0106] Measurement module 410 is used to measure the distance information corresponding to each pixel in the real-world image captured by the camera module through the ranging module;

[0107] Comparison module 420 is used to compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device, and obtain the comparison result corresponding to each pixel.

[0108] The fusion module 430 is used to fuse the virtual image to be displayed and the real-world image according to the comparison results corresponding to each pixel, so as to obtain a fused image and display the fused image.

[0109] In one embodiment, the device 400 further includes a detection module (not shown).

[0110] The detection module is used to detect whether the head-mounted display device is in video perspective mode when the user is wearing the head-mounted display device, before the measurement module 410 obtains the distance information corresponding to each pixel in the real-world image captured by the camera module through the ranging module; and to activate the ranging module when the head-mounted display device is in video perspective mode.

[0111] In one embodiment, the fusion module 430 is specifically configured to: determine the target display rule for any pixel in the real-world image and the corresponding pixel in the virtual image to be displayed based on the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed; fuse the any pixel in the real-world image and the corresponding pixel in the virtual image to be displayed according to the target display rule to obtain a fused image; and display the fused image.

[0112] In one embodiment, the fusion module 430 is specifically configured to: determine the target display rule as displaying any pixel in the real-world image if the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is less than the virtual image distance of the corresponding pixel in the virtual image to be displayed; determine the target display rule as displaying any pixel in the real-world image if the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is greater than the virtual image distance of the corresponding pixel in the virtual image to be displayed; and determine the target display rule as displaying either any pixel in the real-world image or the corresponding pixel in the virtual image to be displayed if the comparison result between the distance information corresponding to any pixel in the real-world image and the virtual image distance of the corresponding pixel in the virtual image to be displayed is equal to the virtual image distance of the corresponding pixel in the virtual image to be displayed.

[0113] In one embodiment, the device 400 further includes a calibration module (not shown).

[0114] The calibration module is used to acquire the inertial data collected by the inertial measurement unit; and to locate the head-mounted display device using the inertial data and the real-world image.

[0115] In one embodiment, the measurement module 410 is further configured to measure the distance information corresponding to each pixel in the new real-world image acquired by the camera module by the ranging module after the fusion module 430 fuses the virtual image to be displayed and the real-world image according to the comparison results corresponding to each pixel, obtains the fused image and displays the fused image.

[0116] In one embodiment, the ranging module is a depth camera, and the number of the depth cameras is greater than or equal to 1 and less than or equal to 4.

[0117] According to embodiments of this disclosure, a head-mounted display device can acquire real-world images of the user's scene using its own camera module, measure the distance information corresponding to each pixel in the real-world image using its own ranging module, and compare the distance information corresponding to each pixel in the real-world image with the virtual image distance of the corresponding pixel in the virtual image to be displayed on the head-mounted display device to obtain the comparison result for each pixel. Then, based on the comparison results for each pixel, the virtual image to be displayed and the real-world image are fused to obtain and display a fused image. In other words, the head-mounted display device can measure the distance information of objects existing in the real world during video perspective and can accurately determine the occlusion relationship between the virtual image and the real-world image, making the display effect of the head-mounted display device more realistic.

[0118] <Equipment Example>

[0119] Figure 5 This is a schematic diagram of the hardware structure of a head-mounted display device according to one embodiment. Figure 5 As shown, the head-mounted display device 500 includes a ranging module 510 and a camera module 520, and also includes a processor 530 and a memory 540.

[0120] The memory 540 can be used to store executable computer instructions.

[0121] The processor 530 can be used to execute the image display method according to the method embodiments of this disclosure, under the control of the executable computer instructions.

[0122] The head-mounted display device 500 can be as follows: Figure 1 The head-mounted display device 1000 shown may also be a device with other hardware structures, and is not limited herein. The head-mounted display device 500 may be, for example, a VR device, a MR device, etc., and is not limited in this embodiment.

[0123] In another embodiment, the head-mounted display device 500 may include the above-mentioned image display control device 400.

[0124] In one embodiment, each module of the above image display device 400 can be implemented by the processor 530 running computer instructions stored in the memory 540.

[0125] Computer-readable storage media

[0126] This disclosure also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, perform the image display method provided in this disclosure.

[0127] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0130] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0131] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0135] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. An image display method characterized by, The application is applied to a head-mounted display device, the head-mounted display device comprises a camera module and a ranging module, and the method comprises the following steps: measuring distance information corresponding to each pixel point in a real-world image collected by the camera module by the ranging module; comparing the distance information corresponding to each pixel point in the real-world image with the virtual image distance of the corresponding pixel point in a to-be-displayed virtual image of the head-mounted display device to obtain a comparison result corresponding to each pixel point; fusing the to-be-displayed virtual image and the real-world image according to the comparison result corresponding to each pixel point to obtain a fused image and display the fused image.

2. The method of claim 1, wherein, Before the step of measuring distance information corresponding to each pixel point in a real-world image collected by the camera module by the ranging module, the method further comprises the following steps: detecting whether the head-mounted display device is in a video see-through mode in the case that a user wears the head-mounted display device; starting the ranging module in the case that the head-mounted display device is in the video see-through mode.

3. The method of claim 1, wherein, The step of fusing the to-be-displayed virtual image and the real-world image according to the comparison result corresponding to each pixel point to obtain a fused image and display the fused image comprises the following steps: determining a target display rule of any pixel point in the real-world image and the corresponding pixel point in the to-be-displayed virtual image according to the comparison result between the distance information corresponding to any pixel point in the real-world image and the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image; fusing any pixel point in the real-world image and the corresponding pixel point in the to-be-displayed virtual image according to the target display rule to obtain a fused image; displaying the fused image.

4. The method of claim 3, wherein, The step of determining a target display rule of any pixel point in the real-world image and the corresponding pixel point in the to-be-displayed virtual image according to the comparison result between the distance information corresponding to any pixel point in the real-world image and the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image comprises the following steps: in the case that the comparison result between the distance information corresponding to any pixel point in the real-world image and the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image is that the distance information corresponding to any pixel point in the real-world image is smaller than the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image, determining that the target display rule is to display any pixel point in the real-world image; in the case that the comparison result between the distance information corresponding to any pixel point in the real-world image and the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image is that the distance information corresponding to any pixel point in the real-world image is greater than the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image, determining that the target display rule is to display the corresponding pixel point in the to-be-displayed virtual image. The comparison result between the distance information corresponding to any pixel point in the real-world image and the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image is that the distance information corresponding to any pixel point in the real-world image is equal to the virtual image distance of the corresponding pixel point in the to-be-displayed virtual image, and it is determined that the target display rule is to display any pixel point in the real-world image or the corresponding pixel point in the to-be-displayed virtual image.

5. The method of claim 1, wherein, The head-mounted display device further comprises an inertial measurement unit, and the method further comprises: acquiring inertial data collected by the inertial measurement unit; positioning the head-mounted display device by using the inertial data and the real-world image.

6. The method of claim 1, wherein, After the step of fusing the to-be-displayed virtual image and the real-world image according to the comparison result corresponding to each pixel point to obtain a fused image and displaying the fused image, the method further comprises: measuring distance information corresponding to each pixel point in a new real-world image collected by the camera module by using the distance measuring module.

7. The method of claim 1, wherein, The distance measuring module is a depth camera, and the number of the depth cameras is greater than or equal to 1 and less than or equal to 4.

8. An image display device, characterized by comprising: The head-mounted display device comprises a camera module and a distance measuring module, and the device comprises: a measuring module configured to measure distance information corresponding to each pixel point in a real-world image collected by the camera module by using the distance measuring module; a comparing module configured to compare the distance information corresponding to each pixel point in the real-world image with virtual image distance of the corresponding pixel point in a to-be-displayed virtual image of the head-mounted display device to obtain a comparison result corresponding to each pixel point; a fusing module configured to fuse the to-be-displayed virtual image and the real-world image according to the comparison result corresponding to each pixel point to obtain a fused image and display the fused image. 9.A head-mounted display device comprising a camera module and a distance measuring module, and further comprising: a memory configured to store executable computer instructions; a processor configured to execute the image display method according to any one of claims 1-7 under the control of the executable computer instructions. 10.A computer readable storage medium having computer instructions stored thereon, wherein the computer instructions are executed by a processor to perform the image display method according to any one of claims 1-7.

Images