Virtual-real fusion method, display module and related equipment

Abstract

The invention provides a virtual-real fusion method, a display module and related equipment, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining current human eye view field information and real world information and virtual world information of an environment where a user is located; selecting target real world information from the real world information according to the current human eye view field information, wherein the target real world information is real world information of the current human eye view field; processing the target real world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; and adjusting the transparency of each pixel point corresponding to the virtual world information according to the transparency value of each pixel point so as to complete virtual-real fusion. According to the invention, the fusion degree of the virtual object and the real environment can be improved, and the visual perception of the virtual object is more real.

Description

Background Art

[0002] In display scenarios where virtual and real objects are integrated, especially in application scenarios where virtual objects are superimposed on the real environment using OST mode (optical perspective display mode) through smart display devices (such as augmented reality glasses, AR glasses), the integration degree between virtual objects and the real world is often low. This situation makes it difficult for virtual objects to visually fit perfectly with the real world and cannot provide natural and realistic visual effects.

[0003] Due to the limitations of current technology, the lighting, shadows, reflections, etc. of virtual objects and the environmental interactions with surrounding real objects are not realistic enough and often appear inconsistent or inappropriate. In addition, due to the limitations of the display accuracy, perspective effect, and spatial perception of AR devices, there may be certain deviations in the positioning and dynamic interaction of virtual objects, which affects the user's perception of the degree of integration of virtual objects with the real environment.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a virtual-reality fusion method, a display module, and related equipment, which can improve the degree of fusion between virtual objects and the real environment, at least to a certain extent, and make the visual experience of virtual objects more realistic.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a method for merging virtuality and reality is provided, including: obtaining current human eye field of view information, real world information of a user's environment, and virtual world information; selecting target real world information from the real world information according to the current human eye field of view information, wherein the target real world information is the real world information of the current human eye field of view; processing the target real world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; and adjusting the transparency of each pixel point corresponding to the virtual world information according to the transparency value of each pixel point to complete the fusion of virtuality and reality.

[0008] In one embodiment of the present disclosure, the target real-world information and the virtual-world information are processed to obtain the transparency value of each pixel corresponding to the virtual-world information, including: performing pixel-aligned fusion processing on the target real-world information and the virtual-world information to obtain the positional relationship and occlusion relationship of the virtual image corresponding to the virtual world information in the real-world image corresponding to the target real-world information; and determining the transparency value of each pixel corresponding to the virtual world information based on the positional relationship and the occlusion relationship.

[0009] In one embodiment of the present disclosure, based on the positional relationship and the occlusion relationship, the transparency value of each pixel point corresponding to the virtual world information is determined, including: determining the overlapping area between the virtual image and the real-world image according to the positional relationship; and determining the transparency value of each pixel point corresponding to the virtual world information located in the overlapping area according to the occlusion relationship.

[0010] In one embodiment of the present disclosure, the transparency value of each pixel corresponding to the virtual world information located in the overlapping area is determined according to the occlusion relationship, including: if the occlusion relationship is that the virtual image occludes the real world image, the transparency value of each pixel corresponding to the virtual world information in the overlapping area is adjusted to a first value, and the first value is a value that makes the virtual image completely visible; if the occlusion relationship is that the real world image occludes the virtual image, the transparency value of each pixel corresponding to the virtual world information in the overlapping area is adjusted to a second value, and the second value is a value that makes the virtual image completely transparent.

[0011] According to another aspect of the present disclosure, a virtual-reality fusion display module is provided, including: an image acquisition module, used to acquire current human eye field of view information and real-world information of a user's environment, and select target real-world information from the real-world information according to the current human eye field of view information, wherein the target real-world information is the real-world information of the current human eye field of view; an image fusion module, used to obtain virtual world information, and process the target real-world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; an adjustment layer, wherein the adjustment layer is attached to the lens and is composed of a plurality of pixels with controllable transparency; and an adjustment layer control module, used to adjust the transparency of each pixel point of the adjustment layer according to the transparency value of each pixel point to complete virtual-reality fusion.

[0012] In one embodiment of the present disclosure, the image fusion module is further used to perform pixel-aligned fusion processing on the target real-world information and the virtual world information to obtain the positional relationship and occlusion relationship of the virtual image corresponding to the virtual world information in the real-world image corresponding to the target real-world information; based on the positional relationship and the occlusion relationship, determine the transparency value of each pixel point corresponding to the virtual world information.

[0013] According to another aspect of the present disclosure, a virtual-reality fusion device is provided, including: an acquisition module, used to acquire current human eye field of view information, real world information of a user's environment, and virtual world information; a selection module, used to select target real world information from the real world information according to the current human eye field of view information, wherein the target real world information is the real world information of the current human eye field of view; a processing module, used to process the target real world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; and an adjustment module, used to adjust the transparency of each pixel point corresponding to the virtual world information according to the transparency value of each pixel point, so as to complete the virtual-reality fusion.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-mentioned virtual-reality fusion methods by executing the executable instructions.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the virtual-reality fusion methods described above is implemented.

[0016] According to another aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program or computer instructions, wherein the computer program or the computer instructions are loaded and executed by a processor so as to enable a computer to implement any of the virtual-reality fusion methods described above.

[0017] In an embodiment of the present disclosure, the current human eye field of view information, the real world information of the user's environment, and the virtual world information are obtained; the target real world information is selected from the real world information according to the current human eye field of view information, and the target real world information is the real world information of the current human eye field of view; the target real world information and the virtual world information are processed to obtain the transparency value of each pixel corresponding to the virtual world information; the transparency of each pixel corresponding to the virtual world information is adjusted according to the transparency value of each pixel to complete the fusion of virtual and real. The present disclosure selects the target real world information from the real world information according to the current human eye field of view information, and processes the target real world information and the virtual world information to obtain the transparency value of each pixel corresponding to the virtual world information, and adjusts the transparency of each pixel corresponding to the virtual world information according to the transparency value of each pixel to complete the fusion of virtual and real, thereby improving the degree of fusion of virtual objects and the real environment, making the visual experience of virtual objects more realistic.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A schematic diagram showing the OST system architecture in an embodiment of the present disclosure.

[0021] Figure 2 A schematic diagram showing a virtual-reality fusion system structure in an embodiment of the present disclosure is shown.

[0022] Figure 3 A schematic diagram of a virtual-reality fusion display module in an embodiment of the present disclosure is shown.

[0023] Figure 4 A flow chart of a virtual-reality fusion method in an embodiment of the present disclosure is shown.

[0024] Figure 5 A flow chart of a virtual-reality fusion method in another embodiment of the present disclosure is shown.

[0025] Figure 6 A schematic diagram showing a virtual-reality fusion display module in another embodiment of the present disclosure is shown.

[0026] Figure 7 A schematic diagram of a virtual-reality fusion device in an embodiment of the present disclosure is shown.

[0027] Figure 8 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown.

[0028] Fig. 9 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] For ease of understanding, several terms involved in the present disclosure are explained below:

[0035] Ontology knowledge model: In computer and related fields, ontology refers to the basic method of applying ontology, which is the theory and method of abstracting entities in the real world into a set of concepts and relationships between concepts through concept analysis and modeling. The ontology knowledge model is a knowledge model built based on ontology technology.

[0036] See-Through technology enables users to see the real world outside the device. There are two main ways to implement See-Through: optical see-through and video see-through.

[0037] OST (Optical See-Through) system superimposes virtual information onto the real world through transparent optical devices (such as augmented reality glasses or head-mounted displays). Figure 1 As shown, the virtual world information is superimposed on the real world information, thereby achieving virtual-real fusion.

[0038] At present, in the display scenes that combine virtual and real, especially through smart display devices (such as AR glasses) using the OST mode to superimpose virtual objects (such as virtual items) in real scenes, due to the transparency control of virtual images, the appearance of virtual objects is not real enough, and the degree of integration between real scenes and virtual objects needs to be improved.

[0039] Based on this, the present disclosure provides a virtual-reality fusion method, which is applied to virtual-reality fusion display scenes, for example, on AR devices. The present disclosure selects target real-world information from real-world information based on the current human eye field of view information, and processes the target real-world information and virtual world information to obtain the transparency value of each pixel corresponding to the virtual world information, and adjusts the transparency of each pixel corresponding to the virtual world information according to the transparency value of each pixel to complete the virtual-reality fusion. In the OST mode, the present disclosure controls the transparency of virtual objects through the fusion processing of real scenes and virtual images and the controllable transparency of pixels, thereby improving the degree of fusion with the real scene and making the visual experience of virtual objects more realistic.

[0040] It should be pointed out that, in the absence of conflict, the embodiments of the present disclosure and the technical features therein may be combined with each other.

[0041] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0042] Figure 2 A schematic diagram of a virtual-reality fusion system structure in an embodiment of the present disclosure is shown, and the system can apply the virtual-reality fusion method or virtual-reality fusion device in each embodiment of the present disclosure.

[0043] like Figure 2As shown, the system architecture 200 may include a terminal device 201 having a virtual-reality fusion display module. The terminal device 201 is configured with the virtual-reality fusion display module, and the virtual-reality fusion display module is used to enable the user to observe the virtual-reality fusion picture by using the terminal device 201.

[0044] The terminal device 201 can be various electronic devices, including but not limited to AR devices, wearable devices, augmented reality devices, virtual reality devices, etc.

[0045] Among them, Figure 3 As shown, a virtual-reality fusion display module provided by the present disclosure may include: an image acquisition module 301 , an image fusion module 302 , an adjustment layer 303 and an adjustment layer control module 304 .

[0046] The image acquisition module 301 can be used to collect the current human eye field of view information and the real-world information of the user's environment, and select the target real-world information from the real-world information based on the current human eye field of view information, and the target real-world information is the real-world information of the current human eye field of view. It should be noted that the image acquisition module 301 performs image acquisition on the real-world information, and the collected information is consistent with the scene information seen by the human eye through the lens. In other words, the image acquisition module 301 selects the target real-world information from the real-world information based on the current human eye field of view information, and the target real-world information is the real-world information of the current human eye field of view, and the real-world information of the current human eye field of view is information that is consistent with the scene information seen by the human eye through the lens.

[0047] The image fusion module 302 can be used to obtain virtual world information, and process the target real world information and the virtual world information to obtain the transparency value of each pixel corresponding to the virtual world information. Exemplarily, the image fusion module 302 can also be used to perform pixel alignment fusion processing on the target real world information and the virtual world information to obtain the position relationship and occlusion relationship of the virtual image corresponding to the virtual world information in the real world image corresponding to the target real world information; based on the position relationship and occlusion relationship, determine the transparency value of each pixel corresponding to the virtual world information.

[0048] It should be noted that the image fusion module 302 fuses the collected image information of the real world (such as the target real world information) and the virtual world (such as the virtual world information), mainly to determine the position of the virtual image in the real world image and its occlusion relationship with the objects in the real world, etc. The virtual-real fusion display module disclosed in the present invention may also include a virtual information optical engine for completing optical projection according to the virtual world information generated by the system.

[0049] The adjustment layer 303 is attached to the lens and is composed of a number of pixels with controllable transparency. It should be noted that the lens can transmit light from the real world, so that the user can directly see the real world scene; it can also reflect virtual world optical information, so that the user can see the projected virtual world information; and it can transmit the current human eye field information to the image acquisition module 301.

[0050] The adjustment layer control module 304 can be used to adjust the transparency of each pixel of the adjustment layer according to the transparency value of each pixel to complete the virtual-real fusion. Exemplarily, the transparency of each pixel on the adjustment layer 303 is controlled by the adjustment layer control module 304, and the reflection of the projected light is completed according to the transparency value of the pixel in cooperation with the lens. For example, the pixel with a transparency value of 0 does not reflect, the pixel with a transparency value of 100 fully reflects, and the pixel in between completes the light reflection according to the transparency value.

[0051] The virtual-reality fusion display module disclosed in the embodiment of the present disclosure adds an adjustment layer with controllable pixel transparency on the lens to control the transparency of the virtual object after projection, and obtains the position relationship and occlusion relationship by fusing the target real-world information and the virtual-world information. Based on the position relationship and the occlusion relationship, the transparency value of each pixel corresponding to the virtual-world information is determined, and the transparency of the pixel is adjusted by the determined transparency value, thereby improving the degree of fusion between the virtual object and the real scene, making the visual experience of the virtual object more realistic.

[0052] Those skilled in the art will know that Figure 1 The number of terminal devices in the example is only illustrative, and any number of terminal devices may be provided according to actual needs. This embodiment of the present disclosure does not limit this.

[0053] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0054] A virtual-reality fusion method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device equipped with a virtual-reality fusion display module.

[0055] Figure 4 A flow chart of a virtual-real fusion method in an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the virtual-reality fusion method provided in the embodiment of the present disclosure may include the following S401 to S404.

[0056] S401, obtaining current human eye field information, real world information and virtual world information of the user's environment.

[0057] In the disclosed embodiment, the current human eye field of view information is the visual range that the human eye can perceive at a specific moment. Real world information is real-time data from the physical environment, which may include perception information such as images, sounds, positions, depths, and motions. Virtual world information is information used to virtualize virtual objects. It should be noted that the current human eye field of view information may also include visual field information. For example, the visual field information may include visual resolution, color perception, and the central and peripheral parts of visual perception. In order to further improve the accuracy of fusion, visual field information may be added to the current human eye field of view information.

[0058] S402, selecting target real-world information from the real-world information according to the current human eye field of view information, where the target real-world information is the real-world information of the current human eye field of view.

[0059] In the embodiment of the present disclosure, the target real-world information is information that is consistent with the scene information seen by the human eye through the lens. For example, the real-world information is a set {S 1 , S 2 , S 3 , S 4 ,……,S n}, n is a positive integer, and each element in the set represents the real-world information that a person can see at a certain angle. Find the target real-world information that matches the current human eye field information from the set. For example, the target real-world information is S 1 That is to say, in the real-world information of multiple perspectives, the real-time information with the same perspective as the current person is found according to the current person's perspective (current human eye field information). It can also be said that the target real-world information is a subset of the real-world information.

[0060] S403, processing the target real world information and the virtual world information to obtain the transparency value of each pixel corresponding to the virtual world information.

[0061] The disclosed embodiments do not limit how to determine the transparency value based on the target real world information and the virtual world information. For example, the transparency value is determined based on the target real world information and the virtual world information through a machine learning model. For another example, the virtual world information and the target real world information are fused by image resolution to determine the transparency value of each pixel.

[0062] S404, adjusting the transparency of each pixel corresponding to the virtual world information according to the transparency value of each pixel, so as to achieve virtual-real fusion.

[0063] The disclosed embodiment adjusts the transparency of each pixel on the adjustment layer through the transparency value determined by the target real world information and the virtual world information, which can enhance the effect of virtual-real fusion.

[0064] The disclosed embodiment selects target real-world information from real-world information according to the current human eye field information, processes the target real-world information and virtual-world information, obtains the transparency value of each pixel corresponding to the virtual-world information, and adjusts the transparency of each pixel corresponding to the virtual-world information according to the transparency value of each pixel to complete the fusion of virtual and real, thereby improving the degree of fusion between virtual objects and the real environment, and making the visual experience of virtual objects more realistic.

[0065] The present disclosure is described below by means of exemplary embodiments.

[0066] In an exemplary embodiment, if Figure 5 As shown, the virtual-reality fusion method provided by the present disclosure processes the target real-world information and the virtual-world information, and obtains the transparency value of each pixel point corresponding to the virtual-world information, including the following S4031 and S4032.

[0067] S4031, performing pixel alignment fusion processing on the target real-world information and the virtual-world information to obtain the position relationship and occlusion relationship of the virtual image corresponding to the virtual-world information in the real-world image corresponding to the target real-world information.

[0068] In the disclosed embodiment, the virtual world information and the target real world information are fused by image resolution, pixel alignment is completed, and the positional relationship of the virtual image corresponding to the virtual world information in the real world image corresponding to the target real world information and the occlusion relationship between the virtual image and the real world image are determined, so as to set the transparency according to the actual application scenario and specific application experience. It should be noted that the positioning of the virtual object is achieved through pixel alignment, thereby reducing deviation and making the fusion more realistic. The pixel is the pixel presented by the display system.

[0069] S4032: Determine the transparency value of each pixel corresponding to the virtual world information based on the position relationship and the occlusion relationship.

[0070] The embodiments of the present disclosure do not specifically limit how to determine the transparency value based on the position relationship and the occlusion relationship.

[0071] Exemplarily, determining the transparency value of each pixel corresponding to the virtual world information based on the position relationship and the occlusion relationship may include the following steps A1 and A2.

[0072] Step A1, determining the overlapping area between the virtual image and the real-world image according to the position relationship.

[0073] In the embodiment of the present disclosure, the overlapping area is the area where the virtual image overlaps with the real-world image.

[0074] Step A2: determining the transparency value of each pixel corresponding to the virtual world information located in the overlapping area according to the occlusion relationship.

[0075] In one embodiment, determining the transparency value of each pixel corresponding to the virtual world information located in the overlapping area according to the occlusion relationship may include the following steps A21 and A22.

[0076] Step A21, if the occlusion relationship is that the virtual image occludes the real world image, the transparency value of each pixel corresponding to the virtual world information in the overlapping area is adjusted to a first value, and the first value is a value that makes the virtual image completely visible.

[0077] In the disclosed embodiment, the first value is the maximum transparency value. If the occlusion relationship is that the virtual image occludes the real-world image, the transparency value of the virtual image is determined to be the maximum value, so that the virtual image (virtual object) visually occludes the real-world image in the overlapping area.

[0078] Step A22, if the occlusion relationship is that the real world image occludes the virtual image, the transparency value of each pixel corresponding to the virtual world information in the overlapping area is adjusted to a second value, and the second value is a value that makes the virtual image completely transparent.

[0079] In the disclosed embodiment, the second value is the minimum transparency value. If the occlusion relationship is that the real-world image occludes the virtual image, the transparency value of the virtual image is determined to be the minimum value, so that the real-world image in the overlapping area visually occludes the virtual image.

[0080] The disclosed embodiment sets virtual objects that block real-world images to be fully visible, and sets blocked virtual objects to be fully transparent, thereby deeply integrating virtual objects with the real environment, achieving seamless integration of virtual and reality, and making the visual experience of virtual objects more realistic.

[0081] In another embodiment, when the virtual object is a transparent object, the transparency value of the corresponding pixel point can be adjusted according to the transparency of the transparent object, so that the transparent object does not completely block the real-world image, and the real-world image visually observed through the virtual object is slightly deformed or blurred, but the blocked real-world image is still visible, thereby further making the visual experience of the virtual object more realistic.

[0082] For example, when the virtual object is a glass with a smooth surface, the transparency value is adjusted to 30. When the virtual object is a glass with a frosted surface, the transparency value is adjusted to 60. It should be noted that the greater the transparency value, the weaker the transparency.

[0083] The present disclosure is described below through a specific embodiment.

[0084] Figure 6 A schematic diagram showing a virtual-reality fusion display module in another embodiment of the present disclosure is shown, Figure 6 The virtual-real fusion method provided by the present disclosure includes the following steps.

[0085] ①, the lens transmits the current human eye field information to the image acquisition module 301.

[0086] ②, the image acquisition module 301 acquires the real-world information of the user's environment, and selects target real-world information from the real-world information according to the current human eye field information, where the target real-world information is the real-world information of the current human eye field.

[0087] ③. The image acquisition module 301 sends the acquired target real-world information to the image fusion module 302.

[0088] ④. The virtual information optical engine projects the virtual world information to the image fusion module 302.

[0089] ⑤, the image fusion module 302 fuses the virtual world information and the target real world information according to the current system image resolution, and completes the pixel alignment of the two types of image information. It also determines the position relationship of the virtual image corresponding to the virtual world information in the real world image corresponding to the target real world information and its occlusion relationship with the objects in the real world; the image fusion module 302 transmits the transparency setting requirements (such as transparency value) of each pixel point of the virtual world information to the adjustment layer control module 304.

[0090] ⑥, the adjustment layer control module 304 completes the transparency control of the corresponding pixel points of the adjustment layer 303 according to the transparency requirements of each pixel point.

[0091] ⑦, the image fusion module 302 projects the virtual information optical engine content to the lens, and the lens completes the reflection of the light according to the transparency value of the pixels of the adjustment layer 303.

[0092] The present disclosure proposes a system framework for realizing transparency control of virtual objects in OST mode, and a realization method and process for controlling the transparency of virtual objects through fusion processing of real scenes and virtual images and an adjustment layer with controllable pixel transparency. In OST mode, the real scene and virtual image are fused, and an adjustment layer with controllable pixel transparency is added to the lens to control the transparency of the virtual object after projection, thereby improving the degree of fusion of the virtual object and the real scene, making the visual experience of the virtual object more realistic.

[0093] Based on the same inventive concept, the present disclosure also provides a virtual-reality fusion device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0094] Figure 7 A schematic diagram of a virtual-real fusion device in an embodiment of the present disclosure is shown. Figure 7 As shown, the virtual-real fusion device may include an acquisition module 71, a selection module 72, a processing module 73, and an adjustment module 74. The acquisition module 71 may be used to acquire the current human eye field of view information, the real world information of the user's environment, and the virtual world information; the selection module 72 may be used to select target real world information from the real world information according to the current human eye field of view information, and the target real world information is the real world information of the current human eye field of view; the processing module 73 may be used to process the target real world information and the virtual world information to obtain the transparency value of each pixel point corresponding to the virtual world information; the adjustment module 74 may be used to adjust the transparency of each pixel point corresponding to the virtual world information according to the transparency value of each pixel point, so as to complete the virtual-real fusion.

[0095] In one embodiment, the processing module 73 can also be used to perform pixel-aligned fusion processing on the target real-world information and the virtual world information to obtain the positional relationship and occlusion relationship of the virtual image corresponding to the virtual world information in the real-world image corresponding to the target real-world information; based on the positional relationship and occlusion relationship, determine the transparency value of each pixel point corresponding to the virtual world information.

[0096] In one embodiment, the processing module 73 may also be used to determine the overlapping area between the virtual image and the real world image according to the position relationship; and determine the transparency value of each pixel corresponding to the virtual world information located in the overlapping area according to the occlusion relationship.

[0097] In one embodiment, the processing module 73 may also be used to adjust the transparency value of each pixel point corresponding to the virtual world information in the overlapping area to a first value if the occlusion relationship is that the virtual image occludes the real-world image, and the first value is a value that makes the virtual image completely visible; if the occlusion relationship is that the real-world image occludes the virtual image, then the transparency value of each pixel point corresponding to the virtual world information in the overlapping area is adjusted to a second value, and the second value is a value that makes the virtual image completely transparent.

[0098] The virtual-reality fusion device disclosed in the embodiment of the present disclosure selects target real-world information from the real-world information according to the current human eye field of view information, and processes the target real-world information and the virtual world information to obtain the transparency value of each pixel point corresponding to the virtual world information, and adjusts the transparency of each pixel point corresponding to the virtual world information according to the transparency value of each pixel point to complete the virtual-reality fusion, thereby improving the degree of fusion of virtual objects and the real environment, and making the visual experience of virtual objects more realistic.

[0099] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0100] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0102] Among them, the storage unit stores a program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can execute the following steps of the above method embodiment: obtain the current human eye field of view information, the real world information of the user's environment, and the virtual world information; select the target real world information from the real world information according to the current human eye field of view information, and the target real world information is the real world information of the current human eye field of view; process the target real world information and the virtual world information to obtain the transparency value of each pixel corresponding to the virtual world information; adjust the transparency of each pixel corresponding to the virtual world information according to the transparency value of each pixel to complete the virtual and real fusion.

[0103] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0104] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0105] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0106] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0107] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0108] In the disclosed exemplary embodiments, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. Fig. 9 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Fig. 9 As shown, the computer-readable storage medium 900 stores a program product capable of implementing the above method of the present disclosure.

[0109] In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present disclosure described in the above "Specific Implementation Methods" section of this specification.

[0110] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0112] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0113] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0114] The embodiment of the present disclosure provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the virtual-reality fusion method provided in various optional ways in any embodiment of the present disclosure.

[0115] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0116] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0117] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0118] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope of the present disclosure is indicated by the appended claims.

Claims

1. A virtual-real fusion method, characterized in that: include: Obtain the current human eye field information, the real world information of the user's environment, and the virtual world information; Selecting target real-world information from the real-world information according to the current human eye field information, the target real-world information being the real-world information of the current human eye field; Processing the target real world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; The transparency of each pixel point corresponding to the virtual world information is adjusted according to the transparency value of each pixel point to achieve virtual-real fusion.

2. The method according to claim 1, characterized in that Processing the target real world information and the virtual world information to obtain the transparency value of each pixel corresponding to the virtual world information includes: Performing pixel alignment fusion processing on the target real-world information and the virtual-world information to obtain a position relationship and an occlusion relationship of a virtual image corresponding to the virtual-world information in a real-world image corresponding to the target real-world information; Based on the position relationship and the occlusion relationship, a transparency value of each pixel point corresponding to the virtual world information is determined.

3. The method according to claim 2, characterized in that Determining the transparency value of each pixel point corresponding to the virtual world information based on the position relationship and the occlusion relationship includes: Determining an overlapping area between the virtual image and the real-world image according to the positional relationship; According to the occlusion relationship, the transparency value of each pixel corresponding to the virtual world information located in the overlapping area is determined.

4. The method according to claim 3, characterized in that: Determining the transparency value of each pixel corresponding to the virtual world information located in the overlapping area according to the occlusion relationship includes: If the occlusion relationship is that the virtual image occludes the real-world image, adjusting the transparency value of each pixel corresponding to the virtual world information in the overlapping area to a first value, wherein the first value is a value that makes the virtual image completely visible; If the occlusion relationship is that the real-world image occludes the virtual image, the transparency value of each pixel corresponding to the virtual world information in the overlapping area is adjusted to a second value, where the second value is a value that makes the virtual image completely transparent.

5. A virtual-reality fusion display module, characterized in that: include: An image acquisition module, used to acquire current human eye field information and real-world information of the user's environment, and select target real-world information from the real-world information according to the current human eye field information, wherein the target real-world information is the real-world information of the current human eye field; An image fusion module is used to obtain virtual world information, and process the target real world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; An adjustment layer, which is attached to the lens and is composed of a number of pixels with controllable transparency; The adjustment layer control module is used to adjust the transparency of each pixel point of the adjustment layer according to the transparency value of each pixel point to achieve virtual-real fusion.

6. The virtual-reality fusion display module according to claim 5, characterized in that: The image fusion module is further used to perform pixel-aligned fusion processing on the target real-world information and the virtual-world information to obtain a position relationship and an occlusion relationship of the virtual image corresponding to the virtual-world information in the real-world image corresponding to the target real-world information; Based on the position relationship and the occlusion relationship, a transparency value of each pixel point corresponding to the virtual world information is determined.

7. A virtual-real fusion device, characterized in that: include: An acquisition module is used to obtain the current human eye field information, the real world information of the user's environment, and the virtual world information; A selection module, configured to select target real-world information from the real-world information according to the current human eye field of view information, wherein the target real-world information is the real-world information of the current human eye field of view; A processing module, used to process the target real world information and the virtual world information to obtain a transparency value of each pixel point corresponding to the virtual world information; The adjustment module is used to adjust the transparency of each pixel point corresponding to the virtual world information according to the transparency value of each pixel point to achieve virtual and real fusion.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the virtual-reality fusion method described in any one of claims 1 to 4 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual-reality fusion method described in any one of claims 1 to 4 is implemented.

10. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, and when the computer instructions are executed by a processor, they are operating instructions for implementing the virtual-reality fusion method described in any one of claims 1 to 4.

Images