Virtual and real scene fusion method, device, electronic device and storage medium

Abstract

The present application discloses a method, device, electronic device and storage medium for fusing virtual and real vision. The method includes: constructing a virtual scene based on a real scene, wherein the virtual scene includes a virtual entity and a twin corresponding to the real entity in a one-to-one manner; obtaining the posture information of the real entity in real time and synchronizing the posture information of the twin; obtaining the posture information of the real viewpoint, and obtaining an initial virtual vision based on the posture information of the real viewpoint and the virtual scene; eliminating the invisible part of the virtual entity and the twin in the initial virtual vision to obtain a target virtual vision; and obtaining a fused vision based on the target virtual vision. The present application eliminates the obscured part of the virtual entity based on the twin of the real entity, and then fuses it with the real vision. The movement of the virtual entity is restricted by the real scene, which effectively ensures the dynamic consistency of the virtual vision and the real vision, and ensures the position accuracy of the virtual entity in the fused vision and the accuracy of the fused depth information.

Description

Technical Field

[0001] The present application belongs to the field of virtual and real vision technology, and specifically relates to a method, device, electronic device and storage medium for fusing virtual and real vision. Background Art

[0002] Virtual-real scene fusion refers to the process of fusing real and virtual scenes to form a fused scene and then visualizing the fused scene. Existing virtual-real scene fusion technologies directly overlay virtual entities onto the real scene. The movement of virtual entities is not restricted by the real scene. During the fusion process for moving scenes, the dynamic consistency of the virtual and real scenes cannot be guaranteed, affecting the positional accuracy of virtual entities in the final fused scene and the accuracy of the fused depth information. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, electronic device and storage medium for fusing virtual and real vision to solve the technical problem in the prior art of directly superimposing virtual vision on real vision and failing to ensure dynamic consistency between the virtual vision and real vision in the process of virtual vision and real vision for moving scenes.

[0004] To achieve the above objectives, the present application provides, in a first aspect, a method for fusing virtual and real scenes, comprising:

[0005] Constructing a virtual scene based on a real scene, wherein the real scene includes a real entity, and the virtual scene includes a virtual entity and a twin corresponding to the real entity in a one-to-one manner;

[0006] Acquire the pose information of the real entity in real time and synchronize the pose information of the twin;

[0007] Acquiring position information of a real viewpoint, and obtaining an initial virtual view based on the position information of the real viewpoint and the virtual scene;

[0008] Eliminating the invisible portion of the virtual entity and the twin in the initial virtual vision to obtain a target virtual vision;

[0009] A fused visual scene is obtained based on the target virtual visual scene.

[0010] In one or more embodiments, the step of obtaining an initial virtual view based on the pose information of the real viewpoint and the virtual scene includes:

[0011] Based on the pose information of the real viewpoint, synchronizing a virtual viewpoint having the same pose information in the virtual scene;

[0012] An initial virtual view is obtained based on the pose information of the virtual viewpoint and the virtual scene.

[0013] In one or more embodiments, the step of removing the invisible portion of the virtual entity and the twin from the initial virtual view to obtain the target virtual view includes:

[0014] Traversing each pixel point of the initial virtual scene, and determining whether the virtual entity and the twin exist at the current pixel point traversed;

[0015] If so, determine whether the depth value of the virtual entity at the current pixel point is greater than the depth value of the twin at the current pixel point;

[0016] If so, eliminate the current pixel of the virtual entity;

[0017] After the traversal is completed, the twins are eliminated.

[0018] In one or more embodiments, the real viewpoint is a depth camera, and the step of eliminating the current pixel of the virtual entity is specifically: setting the texture information and depth information of the current pixel of the virtual entity to not be written, and the step of eliminating the twin is specifically: setting the texture information and depth information of the twin to not be written; or,

[0019] The real viewpoint is the human eye, and an optical perspective device is provided on the observation path of the real viewpoint. The step of eliminating the current pixel point of the virtual entity is specifically: setting the texture information of the current pixel point of the virtual entity to not be written, and the step of eliminating the twin is specifically: setting the texture information of the twin to not be written.

[0020] In one or more embodiments, the real viewpoint is a depth camera;

[0021] The step of obtaining a fused visual scene based on the target virtual visual scene includes:

[0022] Obtaining a real view based on the pose information of the real viewpoint and the real scene;

[0023] The real visual scene and the target virtual visual scene are fused to obtain a fused visual scene.

[0024] In one or more embodiments, the step of fusing the real view and the target virtual view to obtain a fused view includes:

[0025] The depth matrix of the real view and the depth matrix of the target virtual view are fused according to the following formula to obtain a fused depth matrix, which is used to describe the depth value of each pixel.

[0026] ,

[0027] Where D f Denotes the fusion depth matrix, D r Represents the depth matrix of the real scene, D v Represents the depth matrix of the target virtual scene, i and j represent the coordinates of the pixel points;

[0028] The pixel matrix of the real scene and the pixel matrix of the target virtual scene are fused according to the following formula to obtain a fused pixel matrix, where the pixel matrix is ​​used to describe the color value of each pixel point:

[0029] ,

[0030] Where, P f Represents the fused pixel matrix, P r Represents the pixel matrix of the real scene, P v Represents the pixel matrix of the target virtual scene, i and j represent the coordinates of the pixel points;

[0031] A fused scene is obtained based on the fused depth matrix and the fused pixel matrix.

[0032] In one or more embodiments, the real viewpoint is a human eye, and an optical perspective device is provided on the observation path of the real viewpoint;

[0033] The step of obtaining a fused visual scene based on the target virtual visual scene includes:

[0034] The target virtual vision is projected onto the optical perspective device, so that the target virtual vision and the real vision are fused into an image in the human eye to obtain a fused vision.

[0035] To achieve the above-mentioned purpose, the second aspect of the present application provides a virtual and real scene fusion device, comprising:

[0036] A virtual scene construction module is used to construct a virtual scene based on a real scene, wherein the real scene includes a real entity, and the virtual scene includes a virtual entity and a twin corresponding to the real entity one by one;

[0037] A synchronization module, used to obtain the posture information of the real entity in real time and synchronize the posture information of the twin;

[0038] A virtual view acquisition module, configured to acquire position information of a real viewpoint and obtain an initial virtual viewpoint based on the position information of the real viewpoint and the virtual scene;

[0039] a virtual view processing module, configured to remove the invisible portion of the virtual entity and the twin from the initial virtual view to obtain a target virtual view;

[0040] The fusion module is used to obtain a fused view based on the target virtual view.

[0041] To achieve the above-mentioned objectives, the third aspect of the present application provides an electronic device, comprising:

[0042] at least one processor; and

[0043] A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to execute the method for fusing virtual and real scenes as described in any one of the above embodiments.

[0044] To achieve the above objectives, the fourth aspect of the present application provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the method for fusing virtual and real scenes as described in any of the above embodiments.

[0045] Different from the prior art, the present invention has the following advantages:

[0046] This application constructs and synchronizes a virtual scene based on a real scene, obtains a virtual vision from the virtual scene, and removes the occluded part of the virtual entity based on the twin of the real entity, and then merges it with the real vision. The movement of the virtual entity is restricted by the real scene, which effectively ensures the dynamic consistency of the virtual vision and the real vision, and ensures the position accuracy of the virtual entity in the fused vision and the accuracy of the fused depth information. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 This is a flowchart of an embodiment of the method for fusing virtual and real scenes of the present application;

[0049] Figure 2 It is a schematic diagram of an implementation method of a virtual scene of the present application;

[0050] Figure 3 yes Figure 1 A schematic flow chart of an implementation method corresponding to S300;

[0051] Figure 4 is a schematic diagram of an implementation method of the initial virtual scene of the present application;

[0052] Figure 5 yes Figure 1 A schematic flow chart of an implementation method corresponding to S400;

[0053] Figure 6 It is a schematic diagram of an embodiment of the target virtual scene of the present application;

[0054] Figure 7 This is a schematic diagram of the effect eliminated in this application;

[0055] Figure 8 is a schematic diagram of an implementation method of the fused vision of the present application;

[0056] Figure 9 This is a schematic diagram of an application scenario of real vision, virtual vision and fused vision in this application;

[0057] Figure 10 This is a structural diagram of an embodiment of the virtual and real vision fusion device of the present application;

[0058] Figure 11 It is a structural diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0060] At present, the fusion method of virtual vision adopts the method of directly superimposing virtual entities onto real vision. In this method, the virtual entity itself is not constrained by the real scene, resulting in the inability to ensure the motion consistency of the virtual entity and the real entity. Especially in the fusion of virtual and real vision during the motion process, there are problems such as position misalignment and depth misalignment between the real entity and the virtual entity, resulting in poor fusion effect.

[0061] In order to solve the above problems, the applicant has developed a new method for fusing virtual and real vision. This method processes the positional relationship and occlusion relationship between virtual entities and real entities in a virtual scene, and realizes virtual and real fusion based on the processed virtual vision, effectively solving the problem of improving the motion consistency between virtual entities and real entities, ensuring fusion accuracy, and improving fusion effect.

[0062] Specifically, see Figure 1 , Figure 1 It is a flowchart of an implementation method of the virtual and real scene fusion method of the present application.

[0063] like Figure 1 As shown, the method includes:

[0064] S100: Build a virtual scene based on the real scene.

[0065] Among them, the real scene includes real entities, and the virtual scene includes virtual entities and twins that correspond one-to-one to the real entities.

[0066] The real scene and the virtual scene are based on a unified coordinate system, and digital twin technology can be used to accurately model all elements of the real scene to ensure consistency between the two.

[0067] Specifically, any modeling engine commonly used in the field, such as Unreal Engine, can be used to construct a virtual scene based on the collected real-world scene information. This will not be discussed in detail here. Users can design virtual entities based on their fusion needs and add them to the virtual scene, thereby obtaining a virtual scene that includes both twins and virtual entities.

[0068] For example, see Figure 2 , Figure 2 This is a schematic diagram of an embodiment of a virtual scene of the present application. Figure 2 The virtual scene includes twins that correspond one-to-one to real entities, as well as virtual entities constructed by users.

[0069] S200: Acquire the pose information of the real entity in real time and synchronize the pose information of the twin.

[0070] After the virtual scene is constructed, the twin in the virtual scene needs to be synchronized with the entity in the real scene at all times. The synchronized posture information can include position, orientation, posture, movement, etc.

[0071] The synchronization of virtual scenes ensures the dynamic consistency of virtual scenes and real scenes, thereby ensuring the dynamic consistency of virtual entities and real entities in subsequent fusion processing.

[0072] S300: Acquire position information of a real viewpoint, and obtain an initial virtual view based on the position information of the real viewpoint and the virtual scene.

[0073] The real viewpoint refers to the view image collection point in the real scene. By obtaining the position, posture, viewing angle, focal length and other posture information of the real viewpoint, we can obtain the image information that can be collected within its field of view, that is, the view.

[0074] In one embodiment, the real viewpoint may be a depth camera, and the pose information of the depth camera may be obtained based on a sensor disposed on the depth camera and preset parameters of the depth camera.

[0075] In another embodiment, the real viewpoint may also be a human eye, and the posture information of the human eye may be obtained based on a sensor arranged on the human head.

[0076] Based on the pose information of the real viewpoint, the image information that can be collected within the field of view can be obtained from the virtual scene, that is, the initial virtual view.

[0077] Specifically, see Figure 3 , Figure 3 yes Figure 1 A flow chart of an implementation method corresponding to S300.

[0078] like Figure 3 As shown, the method for obtaining the initial virtual scene includes:

[0079] S301 : Based on the posture information of the real viewpoint, synchronize virtual viewpoints with the same posture information in the virtual scene.

[0080] S302: Obtain an initial virtual view based on the pose information of the virtual viewpoint and the virtual scene.

[0081] It can be understood that by constructing a virtual viewpoint with the same posture information in the virtual scene, the field of view of the virtual viewpoint can be obtained, and thus the initial virtual view can be obtained.

[0082] For example, see Figure 4 , Figure 4 Schematic diagram of an implementation of the initial virtual scene of this application. Figure 4 As shown, a virtual viewpoint is constructed in the virtual scene, and the initial virtual view is obtained based on the field of view acquisition of the virtual viewpoint.

[0083] S400: Eliminate the invisible parts of the virtual entity and the twin in the initial virtual vision to obtain the target virtual vision.

[0084] In this embodiment, the position and depth relationship between the twins of the virtual entity and the real entity is processed in the initial virtual view, thereby ensuring the motion consistency of the virtual entity and the real entity.

[0085] Specifically, see Figure 5 , Figure 5 yes Figure 1 A flow chart of an implementation method corresponding to S400.

[0086] like Figure 5 As shown, the method for generating the target virtual scene includes:

[0087] S401: traverse each pixel point of the initial virtual scene and determine whether a virtual entity and a twin exist at the current pixel point traversed.

[0088] If yes, then:

[0089] S402: Determine whether the depth value of the virtual entity at the current pixel point is greater than the depth value of the twin at the current pixel point.

[0090] If yes, then:

[0091] S403: Eliminate the current pixel of the virtual entity.

[0092] S404: After the traversal is completed, the twins are eliminated.

[0093] Based on the above steps, the occluded virtual entity can be eliminated from the virtual view, and then the twin can be eliminated to obtain the target virtual view for fusion with the real view.

[0094] For example, see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the target virtual scene of this application. Figure 6 As shown, the occluded virtual entities and twins in the initial virtual view are eliminated to obtain the target virtual view.

[0095] Specifically, the above-mentioned culling method can be achieved by adjusting the simulation engine's parameter drawing settings for objects. Specifically, the simulation engine of the virtual scene supports the drawing of textures and depths of objects. If a certain type of object is set not to write texture information during rendering, in the final picture, such objects will appear as black without texture on the final picture; if the object is set not to write either texture information or depth information during rendering, the object will not appear on the final picture.

[0096] See also Figure 7 , Figure 7 This is a schematic diagram of the culling effect in this application, where a is the original image, b does not write the texture information of the grass, and the grass appears black, and c does not write the texture information and depth information of the grass, and the grass is not displayed.

[0097] Based on the above principles, objects can be culled from the virtual scene through rendering control. Specifically, in one embodiment, when the real viewpoint is a depth camera, the final fused view image needs to be transmitted to the display screen. Therefore, the texture information and depth information of the objects to be culled can be set to not be written. In this case, the objects to be culled in the target virtual view are not displayed. During the final fusion, the culled parts will be replaced by the corresponding parts of the real view, achieving the effect of virtual and real fusion.

[0098] In another embodiment, when the real viewpoint is the human eye, the virtual vision needs to be projected onto an optical lens device such as a translucent lens in front of the human eye. Therefore, the texture information of the object to be eliminated can be set to not be written. At this time, the object to be eliminated in the target virtual vision is displayed in black, and the corresponding position of this part does not emit light when projected onto the optical lens device. The black part finally observed by the human eye presents the translucent state of the translucent lens itself, and then this part can be replaced by the real vision, achieving the effect of virtual and real fusion.

[0099] S500: Obtain a fused vision based on the target virtual vision.

[0100] Since the target virtual vision includes visible virtual entities, a fused vision can be obtained based on the target virtual vision.

[0101] In one embodiment, when the real viewpoint is a depth camera, the fused view needs to be output to a display device to achieve fused visualization.

[0102] At this time, we can first obtain the real view based on the posture information of the real viewpoint and the real scene; then fuse the real view and the target virtual view to obtain the fused view.

[0103] The method of fusing the real view and the target virtual view may include:

[0104] The depth matrix of the real view and the depth matrix of the target virtual view are fused according to the following formula to obtain a fused depth matrix. The depth matrix is ​​used to describe the depth value of each pixel.

[0105] ,

[0106] Where D f Denotes the fusion depth matrix, D r Represents the depth matrix of the real scene, D v Represents the depth matrix of the target virtual scene, i and j represent the coordinates of the pixel points;

[0107] The pixel matrix of the real scene and the pixel matrix of the target virtual scene are fused according to the following formula to obtain a fused pixel matrix. The pixel matrix is ​​used to describe the color value of each pixel.

[0108] ,

[0109] Where, P f Represents the fused pixel matrix, P r Represents the pixel matrix of the real scene, P v Represents the pixel matrix of the target virtual scene, i and j represent the coordinates of the pixel points;

[0110] Based on the fused depth matrix and the fused pixel matrix, a fused view is obtained.

[0111] For example, see Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the fusion vision of the present application. Figure 8 As shown, the real view and the target virtual view are fused to obtain a fused view, which is then visualized.

[0112] In another embodiment, when the real viewpoint is a human eye, an optical perspective device is provided on the observation path of the real viewpoint, and the user observes the real scene through the optical perspective device. Therefore, the target virtual vision can be directly projected onto the optical perspective device, so that when the user observes, the real vision and the target virtual vision can be fused into images in the human eye to obtain a fused vision.

[0113] The optical perspective device may be any device such as a semi-transparent lens that can display images and through which the human eye can observe the external environment.

[0114] Based on the fusion method of the above-mentioned embodiments, a virtual scene is constructed and synchronized based on the real scene, a virtual vision is obtained from the virtual scene, and the occluded part of the virtual entity is eliminated based on the twin of the real entity, and then fused with the real vision. The movement of the virtual entity is restricted by the real scene, which effectively ensures the dynamic consistency of the virtual vision and the real vision, and ensures the position accuracy of the virtual entity in the fused vision and the accuracy of the fused depth information.

[0115] The virtual-real scene fusion methods described in the above embodiments can be applied to simulation training scenarios. For example, a real training ground can be mapped using drone oblique photography, and the site modeling can be performed based on the mapping data to complete the construction of the virtual scene. During the simulation training process, the training entities wear positioning sensors, attitude sensors, motion capture sensors, and other equipment to collect real-time position, attitude, movement, and other data for real-time synchronization of the twin in the virtual scene.

[0116] The training entity obtains real vision data by wearing a camera. The virtual vision acquisition module synchronizes the virtual camera according to the position, posture, field of view and focal length in the real vision data and obtains the virtual vision data.

[0117] The virtual-reality vision fusion module obtains fused vision data by fusing the images in the real vision and the virtual vision, and outputs the fused image to the head-mounted display terminal, realizing the visualization of the fused vision.

[0118] For example, see Figure 9 , Figure 9This is a schematic diagram of the application scenario of real vision, virtual vision and fused vision in this application. Figure 9 As shown in FIG, by fusing the real view and the virtual view, a fused view for simulation training can be obtained.

[0119] This application also provides a virtual and real scene fusion device, please refer to Figure 10 , Figure 10 This is a structural diagram of an embodiment of the virtual and real scene fusion device of the present application. Figure 10 As shown, the device includes a virtual scene construction module 21 , a synchronization module 22 , a virtual vision acquisition module 23 , a virtual vision processing module 24 and a fusion module 25 .

[0120] The virtual scene construction module 21 is used to construct a virtual scene based on a real scene. The real scene includes a real entity, and the virtual scene includes a virtual entity and a twin corresponding to the real entity.

[0121] The synchronization module 22 is used to obtain the pose information of the real entity in real time and synchronize the pose information of the twin;

[0122] The virtual view acquisition module 23 is used to obtain the position information of the real viewpoint and obtain the initial virtual viewpoint based on the position information of the real viewpoint and the virtual scene;

[0123] The virtual view processing module 24 is used to remove the invisible parts of the virtual entity and the twin in the initial virtual view to obtain the target virtual view;

[0124] The fusion module 25 is used to obtain a fused view based on the target virtual view.

[0125] As above Figures 1 to 9 , a method for fusing virtual and real scenes according to an embodiment of this specification is described. The details mentioned in the above description of the method embodiment also apply to the virtual and real scene fusing device according to an embodiment of this specification. The virtual and real scene fusing device described above can be implemented using hardware, software, or a combination of hardware and software.

[0126] This application also provides an electronic device, see Figure 11 , Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application. Figure 11 As shown, the electronic device 30 may include at least one processor 31, a memory 32 (e.g., a non-volatile memory), a storage 33, and a communication interface 34, and the at least one processor 31, the storage 32, the storage 33, and the communication interface 34 are connected together via a bus 35. The at least one processor 31 executes at least one computer-readable instruction stored or encoded in the storage 32.

[0127] It should be understood that the computer executable instructions stored in the memory 32, when executed, cause at least one processor 31 to perform the above combined operations in various embodiments of this specification. Figures 1-9 Describes the various operations and functions.

[0128] In the embodiments of the present specification, the electronic device 30 may include but is not limited to: a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0129] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in software form), which, when executed by a machine, causes the machine to perform the above-mentioned combined embodiments of the present specification. Figures 1-9 Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes for implementing the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can be enabled to read and execute the instructions stored in the readable storage medium.

[0130] In this case, the program code itself read from the machine-readable medium can implement the functions of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0131] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0132] Those skilled in the art will appreciate that the various embodiments disclosed above may be modified and altered in various ways without departing from the essence of the invention. Therefore, the scope of protection of this specification shall be defined by the appended claims.

[0133] It should be noted that not all steps and units in the above processes and system structure diagrams are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or may be implemented by certain components in multiple independent devices.

[0134] In the above embodiments, hardware unit or module can be realized by mechanical means or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. Hardware unit or processor can also include programmable logic or circuit (such as general purpose processor or other programmable processor), can be temporarily set up to complete the corresponding operation by software. Concrete implementation (mechanical means or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0135] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0136] The foregoing description of the present disclosure is provided to enable any person skilled in the art to implement or use the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the general principles herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is intended to be consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for fusing virtual and real scenes, characterized in that: include: Constructing a virtual scene based on a real scene, wherein the real scene includes a real entity, and the virtual scene includes a virtual entity and a twin corresponding to the real entity in a one-to-one manner; Acquire the pose information of the real entity in real time and synchronize the pose information of the twin; Acquiring position information of a real viewpoint, and obtaining an initial virtual view based on the position information of the real viewpoint and the virtual scene; Eliminating the invisible portion of the virtual entity and the twin in the initial virtual vision to obtain a target virtual vision; Based on the target virtual vision, a fused vision is obtained; The step of removing the invisible part of the virtual entity and the twin from the initial virtual scene to obtain the target virtual scene includes: Traversing each pixel point of the initial virtual scene, and determining whether the virtual entity and the twin exist at the current pixel point traversed; If so, determine whether the depth value of the virtual entity at the current pixel point is greater than the depth value of the twin at the current pixel point; If so, eliminate the current pixel of the virtual entity; After the traversal is completed, the twins are eliminated.

2. The fusion method according to claim 1, characterized in that: The step of obtaining an initial virtual view based on the pose information of the real viewpoint and the virtual scene includes: Based on the pose information of the real viewpoint, synchronizing a virtual viewpoint having the same pose information in the virtual scene; An initial virtual view is obtained based on the pose information of the virtual viewpoint and the virtual scene.

3. The fusion method according to claim 1, characterized in that: The real viewpoint is a depth camera, and the step of eliminating the current pixel of the virtual entity is specifically: setting the texture information and depth information of the current pixel of the virtual entity to not be written, and the step of eliminating the twin is specifically: setting the texture information and depth information of the twin to not be written; or, The real viewpoint is the human eye, and an optical perspective device is provided on the observation path of the real viewpoint. The step of eliminating the current pixel point of the virtual entity is specifically: setting the texture information of the current pixel point of the virtual entity to not be written, and the step of eliminating the twin is specifically: setting the texture information of the twin to not be written.

4. The fusion method according to claim 1, characterized in that: The real viewpoint is a depth camera; The step of obtaining a fused visual scene based on the target virtual visual scene includes: Obtaining a real view based on the pose information of the real viewpoint and the real scene; The real visual scene and the target virtual visual scene are fused to obtain a fused visual scene.

5. The fusion method according to claim 4, characterized in that: The step of fusing the real vision and the target virtual vision to obtain a fused vision comprises: The depth matrix of the real view and the depth matrix of the target virtual view are fused according to the following formula to obtain a fused depth matrix, which is used to describe the depth value of each pixel. ， Where D f Denotes the fusion depth matrix, D r Represents the depth matrix of the real scene, D v Represents the depth matrix of the target virtual scene, i and j represent the coordinates of the pixel points; The pixel matrix of the real scene and the pixel matrix of the target virtual scene are fused according to the following formula to obtain a fused pixel matrix, where the pixel matrix is ​​used to describe the color value of each pixel point: ， Where, P f Represents the fused pixel matrix, P r Represents the pixel matrix of the real scene, P v Represents the pixel matrix of the target virtual scene, i and j represent the coordinates of the pixel points; A fused scene is obtained based on the fused depth matrix and the fused pixel matrix.

6. The fusion method according to claim 1, characterized in that: The real viewpoint is a human eye, and an optical perspective device is provided on the observation path of the real viewpoint; The step of obtaining a fused visual scene based on the target virtual visual scene includes: The target virtual vision is projected onto the optical perspective device, so that the target virtual vision and the real vision are fused into an image in the human eye to obtain a fused vision.

7. A device for fusing virtual and real scenes, characterized in that: include: A virtual scene construction module is used to construct a virtual scene based on a real scene, wherein the real scene includes a real entity, and the virtual scene includes a virtual entity and a twin corresponding to the real entity one by one; A synchronization module, used to obtain the posture information of the real entity in real time and synchronize the posture information of the twin; A virtual view acquisition module, configured to acquire position information of a real viewpoint and obtain an initial virtual viewpoint based on the position information of the real viewpoint and the virtual scene; a virtual view processing module, configured to remove the invisible portion of the virtual entity and the twin from the initial virtual view to obtain a target virtual view; A fusion module, configured to obtain a fused view based on the target virtual view; The step of removing the invisible part of the virtual entity and the twin from the initial virtual scene to obtain the target virtual scene includes: Traversing each pixel point of the initial virtual scene, and determining whether the virtual entity and the twin exist at the current pixel point traversed; If so, determine whether the depth value of the virtual entity at the current pixel point is greater than the depth value of the twin at the current pixel point; If so, eliminate the current pixel of the virtual entity; After the traversal is completed, the twins are eliminated.

8. An electronic device, characterized in that: include: at least one processor; as well as A memory storing instructions, wherein when the instructions are executed by the at least one processor, the at least one processor executes the method for fusing virtual and real vision according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that Executable instructions are stored, and when the instructions are executed, the machine executes the virtual and real vision fusion method according to any one of claims 1 to 6.

Images