Virtual-real fusion visual effect display method and related device and system

Abstract

The embodiment of the invention discloses a virtual-real fusion visual effect display method and a related device and system. The method comprises the steps of generating interaction information between virtual and real elements based on ontology knowledge information corresponding to a real scene and a virtual article; the interaction information between the virtual and real elements is mapped to a real scene video to obtain a fusion scene video containing virtual and real element interaction, and the real scene video is generated based on a real scene collected in real time; comparing the fused scene video with the real scene video to obtain difference information between the fused scene video and the real scene video; and inserting a video mask adjustment layer between the fused scene video and the real scene video, and adjusting the transparency of the video mask adjustment layer based on the difference information to display the interaction effect between the virtual article and the real scene. According to the invention, the visual reality sense of superimposing the virtual article in the real space can be improved.

Description

Technical Field

[0001] The present application relates to the field of virtual-reality fusion display technology, and specifically to a virtual-reality fusion visual effect display method and related devices and systems. Background Art

[0002] Virtual-reality fusion technology refers to combining virtual elements with the real environment to create a unique visual experience. This technology allows users to see and interact with virtual elements in the real world, thereby enhancing the perception and interactivity of the real world. Perspective technology refers to presenting virtual elements to users in a realistic way through specific technical means, such as optical perspective or video perspective. Therefore, perspective technology is a way to achieve virtual-reality combined display.

[0003] In the related art, virtual objects are usually superimposed on the real scene in a suspended state, and there is no interaction effect between the objects, resulting in a lack of realism in the virtual and real combined display scene. Therefore, how to improve the visual realism produced by superimposing virtual objects on the real scene is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0004] To solve the above technical problems, the embodiments of the present application respectively provide a virtual-reality fusion visual effect display method, a virtual-reality fusion visual effect display device, a virtual-reality fusion visual effect display system, an electronic device, a computer-readable storage medium and a computer program product.

[0005] According to one aspect of an embodiment of the present application, a method for displaying a virtual-reality fusion visual effect is provided, comprising: generating interaction information between virtual and real elements based on ontological knowledge information corresponding to a real scene and a virtual object; mapping the interaction information between the virtual and real elements to a real scene video to obtain a fused scene video containing interaction between virtual and real elements, wherein the real scene video is generated based on a real scene collected in real time; comparing the fused scene video with the real scene video to obtain difference information between the fused scene video and the real scene video; inserting a video mask adjustment layer between the fused scene video and the real scene video, and adjusting the transparency of the video mask adjustment layer based on the difference information to display the interaction effect between the virtual object and the real scene.

[0006] In another exemplary embodiment, a video mask adjustment layer is inserted between the fused scene video and the real scene video, and the transparency of the video mask adjustment layer is adjusted based on the difference information, including: obtaining a dynamically changing user field of view; and adjusting the transparency of a visible area of ​​the video mask adjustment layer based on the difference information.

[0007] In another exemplary embodiment, the transparency of the visible area of ​​the video mask adjustment layer is adjusted based on the difference information, including: dividing the visible area into multiple sub-areas of different categories according to the difference information; and adjusting the transparency of each sub-area in the visible area based on the category corresponding to each sub-area in the visible area.

[0008] In another exemplary embodiment, each sub-region in the visible area corresponds to one of a difference domain, a transition domain, and a transparent domain; based on the categories corresponding to the sub-regions in the visible area, the transparency of each sub-region in the visible area is adjusted, including: adjusting the sub-region classified as the difference domain to the minimum transparency supported by the video mask adjustment layer; adjusting the sub-region classified as the transparent domain to the maximum transparency supported by the video mask transparent layer.

[0009] In another exemplary embodiment, the ontology knowledge information includes element attribute information and interaction information between elements; the generation of interaction information between virtual and real elements based on the ontology knowledge information corresponding to the real scene and the virtual object includes: performing interaction calculation between the virtual object and the real scene according to the element attribute information and the interaction information between elements contained in the ontology knowledge information, and preset interaction rules, to obtain the interaction information between the virtual and real elements.

[0010] According to one aspect of an embodiment of the present application, a virtual-real fusion visual effect display device is provided, comprising: a generation module, configured to generate interaction information between virtual and real elements based on ontological knowledge information corresponding to a real scene and a virtual object; an interaction module, configured to map the interaction information between the virtual and real elements to a real scene video to obtain a fused scene video containing interaction between virtual and real elements, wherein the real scene video is generated based on a real scene collected in real time; a comparison module, configured to compare the fused scene video with the real scene video to obtain difference information between the fused scene video and the real scene video; a fusion module, configured to insert a video mask adjustment layer between the fused scene video and the real scene video, and adjust the transparency of the video mask adjustment layer based on the difference information to display the interaction effect between the virtual object and the real scene.

[0011] According to one aspect of an embodiment of the present application, a virtual-real fusion visual effect display system is provided, comprising: a spatial ontology knowledge base, used to generate ontology knowledge information corresponding to real scenes and virtual objects; a scene interaction module, used to generate interaction information between virtual and real elements according to interaction rules; an interaction information generation module, used to map the interaction information between the virtual and real elements to a real scene video, so as to generate a fused scene video containing interaction between virtual and real elements; a fusion information analysis module, used to compare the fused scene video and the real scene video, and identify difference information; an adjustment module, used to divide the visible area according to the difference information, and determine the transparency adjustment strategy corresponding to the video mask adjustment layer according to the division result; a video fusion module, used to execute the transparency adjustment strategy, so as to insert a video mask adjustment layer between the fused scene video and the real scene video, and adjust the transparency of the video mask adjustment layer accordingly, so as to display the interaction effect between the virtual object and the real scene.

[0012] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the virtual-reality fusion visual effect display method as described above.

[0013] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the virtual-reality fusion visual effect display method as described above.

[0014] According to one aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the virtual-reality fusion visual effect display method as described above.

[0015] The technical solution provided in the embodiment of the present application generates interaction information between virtual and real elements and maps the interaction information to a real scene video to obtain a fused scene video containing the interaction of virtual and real elements. Then, based on the difference between the fused scene video and the real scene video, the transparency of the video mask adjustment layer inserted between the fused scene video and the real scene video is adjusted to show the interaction effect between the virtual objects and the real scene, thereby enhancing the visual realism of superimposing virtual objects in the real space.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 A schematic diagram of the structure of an exemplary virtual-reality fusion display smart glasses is shown;

[0019] Figure 2 An exemplary architecture diagram of a virtual-reality fusion visual effect display system is shown;

[0020] Figure 3 is a flow chart of a method for displaying a virtual-real fusion visual effect shown in an exemplary embodiment of the present application;

[0021] Figure 4 is a flowchart of a method for displaying a virtual-real fusion visual effect shown in another exemplary embodiment of the present application;

[0022] Figure 5 A schematic diagram showing a process of adjusting the transparency of a visible area of ​​a video mask adjustment layer;

[0023] Figure 6 is a block diagram of a virtual-real fusion visual effect display device shown in an exemplary embodiment of the present application;

[0024] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0028] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0029] First of all, it should be explained that the virtual-reality fusion display scene is used to combine virtual elements with the real environment to create a unique visual experience. It allows users to see and interact with virtual elements in the real world, thereby enhancing the perception and interactivity of the real world.

[0030] Virtual-reality fusion display scenes mainly rely on virtual reality (VR), augmented reality (AR), mixed reality (MR), holographic projection and other technical means. These technologies capture the information in the real scene and integrate it with the virtual information generated by the computer in real time, and finally display it to the user through a display or other visual presentation device. They have the characteristics of strong immersion, strong interactivity and good visualization effect.

[0031] Smart glasses for virtual-reality fusion display are wearable devices that overlay virtual information onto the real world, allowing users to see the fusion of virtual objects and the real environment directly through the glasses. They are usually equipped with micro-projectors, micro-displays, sensors such as gyroscopes and accelerometers, cameras, spatial positioning modules, micro-processing chips and other components, and through the close cooperation between these components, the fusion display of virtual objects and the real environment is realized.

[0032] Virtual-reality fusion display smart glasses can play an important role in many fields such as cultural tourism navigation, game entertainment, and education. For example, users can use virtual-reality fusion display smart glasses to get a more intuitive and convenient navigation experience, improve travel efficiency, and reduce the risk of getting lost; game players can use virtual-reality fusion display smart glasses to get a more immersive gaming experience and feel more real and vivid gaming effects; students can use virtual-reality fusion display smart glasses to see three-dimensional cell structures, historical scenes and other virtual models, so as to have a deeper understanding of relevant knowledge.

[0033] Figure 1FIG. 1 shows a schematic diagram of an exemplary structure of a virtual-real fusion display smart glasses. Figure 1 As shown, the virtual-reality fusion display smart glasses 100 may include a left lens 110a, a right lens 110b, a frame 120, a left temple 130a, and a right temple 130b. The frame 120 may be used to fix the left lens 110a and the right lens 110b. The left temple 130a and the right temple 130b may be used to help the user wear the virtual-reality fusion display smart glasses 100. The left lens 110a and the right lens 110b are used to display the virtual-reality fusion image.

[0034] In addition, the virtual-reality fusion display smart glasses 100 may also include a camera 140, which captures images of the real world and displays them in the glasses to provide intuitive visual information for users. The virtual-reality fusion display smart glasses 100 may also include an ultrasonic sensor, which is used to capture objects and actions in the real world and provide accurate environmental perception capabilities. The virtual-reality fusion display smart glasses 100 may also include an image processing and fusion module to fuse virtual images with images of the real world, and through video overlay technology, seamlessly embed virtual images into images of the real world to achieve seamless switching and interaction between virtual and reality. The virtual-reality fusion display smart glasses 100 may also include an optical display system, which projects the processed image onto the user's retina or displays it through the left lens 110a and / or the right lens 110b. As a result, users can use the virtual-reality fusion display smart glasses 100 to obtain an enhanced display effect of the interaction between real scenes and virtual images.

[0035] It should be noted that this article only uses virtual-reality fusion display smart glasses as an example to introduce the virtual-reality fusion display technology, but it does not mean that the virtual-reality fusion display technology can only be applied in virtual-reality fusion display smart glasses. For example, the virtual-reality fusion display technology can also be applied to devices such as projectors, which is not limited here.

[0036] At present, virtual-reality fusion display scenes mainly use perspective technology, such as optical perspective or video perspective, to present virtual elements to users in a realistic way. However, virtual objects are usually superimposed on the real scene in a suspended state, and there is no interaction effect between objects. As a result, the virtual-reality fusion display scene lacks the real sense of interaction between virtual objects and real scenes.

[0037] In order to enhance the visual realism produced by superimposing virtual objects in real scenes, the embodiments of the present application propose to complete the simulation of the interaction between various elements in the virtual scene, and introduce a dynamic video mask adjustment layer to optimize and process the virtual object images and improve the visualization, thereby enhancing the interaction effect between the virtual objects and the elements in the real scene, thereby enhancing the visual realism of the virtual objects superimposed in the real space.

[0038] See also Figure 2 , Figure 2 FIG. 1 shows an exemplary architecture diagram of a virtual-reality fusion visual effect display system. Figure 2 As shown, the system includes a virtual item information module, a real scene information module, a space ontology knowledge base, a scene information acquisition module, a scene interaction module, an interaction information generation module, a fusion information analysis module, an adjustment module and a video fusion module.

[0039] The real scene information module is used to provide the system with all information about the real scene elements within the current user's visual range. The real scene is the scene that the user can see in the real world, and can also be understood as all information about the real scene elements within the current user's visual range.

[0040] The virtual item information module is used to provide the system with the required information about virtual items.

[0041] The spatial ontology knowledge base is used to generate ontology knowledge information corresponding to real scenes and virtual items. It can also be understood that the spatial ontology knowledge base contains knowledge information of all elements (real elements and virtual elements) in this space, such as the element's own attribute information and the interaction information between elements.

[0042] The spatial ontology knowledge base can be configured with an ontology knowledge model. It can be understood that in computer and related fields, ontology is understood to refer to the basic method of applying ontology, through concept analysis and modeling, abstracting entities in the real world into a set of concepts and relationships between concepts. The ontology knowledge model is a knowledge model built based on ontology technology.

[0043] The scene information acquisition module is used to collect real scene information and generate real-time video. It can be understood that the real-time video is generated based on the real scene and does not contain any virtual elements.

[0044] The scene interaction module is used to generate interaction information between virtual and real elements according to the interaction rules. Specifically, the scene interaction module and the multi-element information in the spatial ontology knowledge base complete the interaction calculation between virtual objects and real scenes to generate interaction information between virtual and real elements. Interaction calculation usually relies on interaction rules, which are usually some prior information. For example, a virtual character sitting on a real sofa will cause the sofa to be depressed and deformed due to heavy pressure.

[0045] The interaction information generation module is used to map the interaction information to the real scene to generate a fusion scene video containing the interaction of virtual and real elements. Specifically, the interaction information generation module generates a simulated interaction video according to the real interaction based on the interaction information between the virtual and real elements. The simulated interaction video contains both virtual elements and real elements and completes the real interaction simulation. For example, when a virtual character sits on a sofa, the sofa is pressed down and deformed.

[0046] The fusion information analysis module is used to compare the fusion scene video and the real scene video and identify the difference information. It can also be understood that the fusion information analysis module synchronously analyzes the two types of video information, the real scene and the virtual-real fusion scene. The difference between the two types of videos is the interaction between the virtual objects and the real scene.

[0047] The adjustment module is used to divide the user's visible area according to the difference information between the fused scene video and the real scene video, and formulate a dynamic adjustment plan for the video mask according to the division structure, that is, the transparency adjustment strategy corresponding to the video mask adjustment layer.

[0048] The video fusion module is used to receive and execute the transparency adjustment strategy to insert a video mask adjustment layer between the fusion scene video and the real scene video, and adjust the transparency of the video mask adjustment layer accordingly to complete the superposition and fusion of the two types of video information, the real scene and the virtual-real fusion scene, and show the interaction effect between the virtual objects and the real scene. That is to say, in the video scene seen by the user, the virtual character sitting on the real sofa will cause the sofa to be depressed and deformed due to heavy pressure, which enhances the visual realism of implanting virtual objects in the real space.

[0049] It should also be noted that the modules included in the virtual-reality fusion visual effect display system illustrated in this embodiment may be software modules, hardware modules, or modules combining software and hardware, which are not limited here. In addition, in actual application scenarios, at least one module may be deployed on the same physical device, which is not limited in this embodiment.

[0050] See also Figure 3 , Figure 3 is a flowchart of a method for displaying a virtual-real fusion visual effect according to an exemplary embodiment of the present application. It should be noted that the method can be Figure 1 The virtual-reality fusion smart glasses 100 shown are specifically implemented so that the user can obtain the visual reality of the fusion of virtual objects and real scenes through the virtual-reality fusion smart glasses 100. Of course, the method can also be executed by other smart devices, such as projectors, etc., and this embodiment is not limited to this.

[0051] like Figure 3As shown, in an exemplary embodiment, the virtual-real fusion visual effect display method includes S310-S340, which are described in detail as follows:

[0052] S310, based on the ontological knowledge information corresponding to the real scene and the virtual object, generate interactive information between the virtual and real elements.

[0053] First of all, it should be noted that ontology knowledge information refers to information obtained through the spatial ontology knowledge base. The spatial ontology knowledge base refers to a highly structured and semantically rich information storage system that contains detailed knowledge information of all elements (including real elements and virtual elements) in a specific space (real space and virtual space). This information covers the attribute information of the elements themselves and the interaction information between elements, that is, the ontology knowledge information includes the attribute information of the elements themselves and the interaction information between elements.

[0054] It is understandable that the attribute information of the element itself may include, for example, basic information, physical attributes, functional attributes, state information, etc., which are not limited here. Basic information may include the unique identifier, name, type, location, size, shape and other information of the element. For real elements, physical attributes may include information such as mass, density, hardness, etc.; for virtual elements, physical attributes may include information such as color, texture, transparency, etc. Functional attributes are used to describe the specific role or ability of an element in a real scene or virtual environment. State information is used to describe the state of an element at a specific moment, such as switch state, temperature, pressure, etc.

[0055] Interaction information between elements can include spatial relationships, dynamic interactions, causal relationships, dependencies, etc., but this is not limited here. Spatial relationships between elements include relative positions, distances, directions, etc., which are crucial for understanding the layout and distribution of elements in space. Dynamic interaction information describes how elements influence and change each other. Causal information reveals the chain of events and logical order between elements. Dependency describes the mutual dependence between elements in terms of function or attributes.

[0056] In some exemplary embodiments, the interaction information between virtual objects and the real scene is calculated based on the element's own attribute information and the interaction information between elements contained in the ontology knowledge information, as well as preset interaction rules, so that the interaction information between virtual and real elements can be obtained.

[0057] It should be understood that based on the ontological knowledge information corresponding to the real scene and the virtual object, the information of the real scene and the virtual object can be obtained respectively, and the preset interaction rules represent the prior knowledge of the interaction between virtual and real elements. For example, when a virtual character sits on a real sofa, the sofa will be pressed down and deformed. Therefore, based on the ontological knowledge information corresponding to the real scene and the virtual object, and the preset prior knowledge representing the interaction between virtual and real elements, the interaction calculation between the virtual object and the real scene can be performed to obtain the interaction information between the virtual and real elements.

[0058] It is also necessary to understand that this embodiment uses the spatial ontology knowledge base to enable the system to more accurately understand objects and events in real scenes, thereby generating more realistic interaction effects between virtual objects and real scenes. This provides users with a more immersive experience, making the performance of virtual objects in real environments more natural and realistic.

[0059] Furthermore, since the spatial ontology knowledge base not only provides basic information about objects but also contains the relationships and rules between them, the system can intelligently handle the interaction between virtual objects and real scenes based on contextual information. For example, the system can identify and avoid collisions between virtual objects and real objects, or dynamically adjust the behavior of virtual objects based on the position and trajectory of real objects.

[0060] Moreover, the spatial ontology knowledge base provides application developers with rich semantic information and contextual support, allowing them to more efficiently develop spatial computing applications with complex interactive functions, which not only shortens the development cycle but also improves the stability and reliability of the application.

[0061] S320, mapping the interaction information between the virtual and real elements to the real scene video to obtain a fused scene video containing the interaction between the virtual and real elements.

[0062] This step generates a simulated interactive video, i.e., a fused scene video, based on the interactive information between virtual and real elements and the real interaction. The fused scene video contains both virtual elements and real elements and completes the interactive simulation. For example, when a virtual character sits on a real sofa, the sofa is pressed down and deformed. S330, compare the fused scene video with the real scene video to obtain the difference information between the fused scene video and the real scene video.

[0063] This step synchronously analyzes and compares the two types of videos, the real scene and the virtual-reality fusion scene, to obtain the difference information between the two types of videos. This difference information is generated by the interaction between the virtual objects and the real scene.

[0064] S340, inserting a video mask adjustment layer between the fused scene video and the real scene video, and adjusting the transparency of the video mask adjustment layer based on the difference information to show the interaction effect between the virtual object and the real scene.

[0065] This step inserts a video mask adjustment layer between the fusion scene video and the real scene video, and adjusts the transparency of the video mask adjustment layer according to the difference information between the fusion scene video and the real scene video, so as to show the interaction effect between the virtual object and the real scene. That is, by inserting a video mask adjustment layer whose transparency can be dynamically adjusted between the real scene and the virtual-real fusion scene, it is possible to show the process of a virtual character sitting on a real sofa and the sofa being sunken and deformed due to heavy pressure, so that the interaction effect produced by superimposing the virtual object on the real scene is more realistic.

[0066] It is understandable that the video mask adjustment layer can also be understood as a video with a specific transparency, and the specific transparency can be adjusted dynamically, which determines which parts of the fused scene video should be displayed. For example, the part that does not differ from the real scene needs to be displayed, so the transparency of the corresponding video mask area is adjusted to be completely transparent, and the part that differs from the real scene is often the part where the virtual object interacts with the real scene. In order to show a more realistic interaction effect, this part can be further divided into a difference part and a transition part. The transparency of the video mask area corresponding to the difference part is adjusted to be completely opaque, and the transparency of the video mask area corresponding to the transition part is between completely opaque and completely transparent, thereby more realistically showing the interaction effect between the virtual object and the real scene.

[0067] Therefore, when generating the fused scene video, this embodiment does not directly display the fused scene video, but continues to synchronously analyze and compare the fused scene video and the real scene video. This is a dynamic process. Each frame of the two types of videos may produce different differences. Therefore, by inserting a video mask adjustment layer between the two accumulated videos and adjusting the transparency of the video mask adjustment layer based on the difference between each frame, thereby dynamically adjusting, such a dynamic process can show the interaction process between virtual objects and real scenes. For example, when a virtual character sits on a real sofa, the sofa is pressed hard and becomes concave and deformed. This virtual fusion visual effect is definitely more realistic than the visual effect of directly displaying the fused scene video.

[0068] Please continue to refer to 4. Figure 4: is a flowchart of a method for displaying a virtual-real fusion visual effect shown in another exemplary embodiment of the present application. This embodiment takes into account that in actual application scenarios, the user's turning of the head or movement will cause a change in the field of vision, so the fusion of virtual objects and real scenes is usually performed dynamically with the dynamically changing user field of vision. Therefore, the process of adjusting the transparency of the video mask adjustment layer based on the difference information between the fused scene video and the real scene video further includes the following steps:

[0069] S410, obtaining a dynamically changing user field of view;

[0070] S420: Adjust the transparency of the visible area of ​​the video mask adjustment layer based on the difference information.

[0071] In the above process, since the user's turning head or moving will cause the field of view to change, it is necessary to dynamically process the characteristics of the information in the field of view in real time to better integrate the virtual elements with the real scene. Therefore, by obtaining the dynamically changing user field of view, the current visible area can be determined, and the transparency of the visible area of ​​the video mask adjustment layer can be adjusted according to the difference information between the fused scene video and the real scene video in the current visible area, so that the display effect of the virtual and real fusion scene that changes dynamically with the user's field of view can be achieved.

[0072] It should be noted that in some exemplary embodiments, since the virtual-reality fusion display smart device is usually equipped with a variety of sensors, such as gyroscopes, accelerometers, cameras, etc., these sensors can sense the user's head movements, position changes, and information about the surrounding environment in real time. For example, the gyroscope and accelerometer can sense the user's head posture and movement, thereby helping the smart device to adjust the position and angle of the virtual object to keep it consistent with the user's line of sight. Therefore, changes in the user's field of vision can be obtained through these sensors.

[0073] In some exemplary embodiments, Figure 5 As shown, the process of adjusting the transparency of the visible area of ​​the video mask adjustment layer includes the following steps:

[0074] S510, dividing the visible area into a plurality of sub-areas of different categories according to the difference information;

[0075] S520: Adjust the transparency of each sub-region in the visible area based on the category corresponding to each sub-region in the visible area.

[0076] The above process divides the visible area of ​​the video mask adjustment layer into multiple sub-areas of different categories according to the difference information between the fused scene video and the real scene video, so as to adjust the transparency of each sub-area accordingly according to the category of each sub-area, so as to more realistically simulate the interactive effect produced by superimposing virtual objects on the real scene.

[0077] Exemplarily, the categories of the sub-regions may include a difference domain, a transition domain, and a transparent domain, and each sub-region corresponds to one of the categories. The difference domain refers to a region where there is a difference between the fused scene video and the real scene video, the transparent domain refers to a region where there is no difference between the fused scene video and the real scene video, and the transition domain is located between the difference domain and the transparent domain, and represents the transition between the difference domain and the transparent domain.

[0078] The sub-region of the difference domain is adjusted to the minimum transparency supported by the video mask adjustment layer, indicating that the sub-region is completely opaque and is used to display the clear boundary of the virtual object. The sub-region of the transparent domain is adjusted to the maximum transparency supported by the video mask transparency layer, indicating that the sub-region is completely opaque. The transparency of the sub-region of the filter domain needs to be adjusted between the minimum transparency and the maximum transparency, so as to achieve the image projection of the virtual object and the smooth transition between the virtual object and the real scene by controlling the transparency of each pixel in the sub-region.

[0079] It should be noted that this embodiment does not limit the method for obtaining the transparency of each pixel in the sub-region of the filter domain. For example, a gradual transparency can be set, such as by setting and adjusting a transparency slider, to gradually transition from complete opacity in the difference domain to complete transparency in the transparent domain. The transparency of each pixel in the transition domain can also be set according to the pixel information such as color and brightness of the corresponding pixel position in the fused scene video and the real scene video to achieve a natural filtering effect.

[0080] Therefore, by reasonably setting the transparency of the transition zone, a smooth transition and fusion between virtual objects and real scenes can be achieved, which enhances the interaction effect between virtual objects and elements in real scenes and improves the visual realism of virtual objects implanted in real space.

[0081] Figure 6 FIG. 1 is a block diagram of a virtual-real fusion visual effect display device shown in an exemplary embodiment of the present application. Figure 6 , the device comprises:

[0082] A generation module 610 is configured to generate interaction information between virtual and real elements based on the ontology knowledge information corresponding to the real scene and the virtual object;

[0083] An interaction module 620 is configured to map the interaction information between virtual and real elements to a real scene video to obtain a fused scene video containing the interaction of virtual and real elements, where the real scene video is generated based on the real scene collected in real time;

[0084] A comparison module 630 is configured to compare the fused scene video with the real scene video to obtain difference information between the fused scene video and the real scene video;

[0085] The fusion module 640 is configured to insert a video mask adjustment layer between the fused scene video and the real scene video, and adjust the transparency of the video mask adjustment layer based on the difference information to show the interaction effect between the virtual object and the real scene.

[0086] In another exemplary embodiment, the fusion module 640 is further configured to perform the following steps:

[0087] Get the dynamically changing user field of view;

[0088] The transparency of the visible area of ​​the video mask adjustment layer is adjusted based on the difference information.

[0089] In another exemplary embodiment, the fusion module 640 is further configured to perform the following steps:

[0090] Dividing the visible area into multiple sub-areas of different categories according to the difference information;

[0091] Based on the categories corresponding to the sub-areas in the visible area, the transparency of the sub-areas in the visible area is adjusted.

[0092] In another exemplary embodiment, each sub-area in the visible area corresponds to one of a difference domain, a transition domain, and a transparent domain; the fusion module 640 is further configured to perform the following steps:

[0093] The sub-region classified as the difference domain is adjusted to the minimum transparency supported by the video mask adjustment layer;

[0094] Adjust the sub-region of the transparent region to the maximum transparency supported by the video mask transparency layer.

[0095] In another exemplary embodiment, the ontology knowledge information includes element attribute information and interaction information between elements; the interaction module 620 is further configured to perform interaction calculations between virtual objects and real scenes based on the element attribute information and interaction information between elements contained in the ontology knowledge information, as well as preset interaction rules, to obtain interaction information between virtual and real elements.

[0096] It should be noted that the virtual-reality fusion visual effect display device provided in the above embodiment and the virtual-reality fusion visual effect display method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0097] The device provided by the above embodiment generates interaction information between virtual and real elements and maps the interaction information to the real scene video to obtain a fused scene video containing the interaction of virtual and real elements, and then adjusts the transparency of the video mask adjustment layer inserted between the fused scene video and the real scene video based on the difference between the fused scene video and the real scene video to show the interaction effect between the virtual objects and the real scene, thereby enhancing the visual realism of superimposing virtual objects in the real space.

[0098] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the virtual-reality fusion visual effect display method provided in the above-mentioned embodiments.

[0099] Figure 7 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0100] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method described in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0101] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0102] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0103] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0104] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0106] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the virtual-reality fusion visual effect display method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0107] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the virtual-reality fusion visual effect display method provided in each of the above embodiments.

[0108] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for displaying virtual-real fusion visual effects, characterized in that: include: Generate interactive information between virtual and real elements based on the ontological knowledge information corresponding to real scenes and virtual objects; Mapping the interaction information between the virtual and real elements to a real scene video to obtain a fused scene video containing the interaction of the virtual and real elements, wherein the real scene video is generated based on a real scene collected in real time; Comparing the fused scene video with the real scene video to obtain difference information between the fused scene video and the real scene video; A video mask adjustment layer is inserted between the fused scene video and the real scene video, and the transparency of the video mask adjustment layer is adjusted based on the difference information to display the interaction effect between the virtual object and the real scene.

2. The method according to claim 1, characterized in that: The inserting a video mask adjustment layer between the fused scene video and the real scene video, and adjusting the transparency of the video mask adjustment layer based on the difference information, comprises: Get the dynamically changing user field of view; The transparency of the visible area of ​​the video mask adjustment layer is adjusted based on the difference information.

3. The method according to claim 2, characterized in that The adjusting the transparency of the visible area of ​​the video mask adjustment layer based on the difference information includes: Dividing the visible area into a plurality of sub-areas of different categories according to the difference information; Based on the categories corresponding to the sub-areas in the visible area, the transparency of the sub-areas in the visible area is adjusted.

4. The method according to claim 3, characterized in that Each sub-area in the visible area corresponds to one of a difference domain, a transition domain, and a transparent domain; and adjusting the transparency of each sub-area in the visible area based on the category corresponding to each sub-area in the visible area includes: Adjusting the sub-region classified as the difference domain to the minimum transparency supported by the video mask adjustment layer; The sub-region classified as the transparent region is adjusted to the maximum transparency supported by the video mask transparency layer.

5. The method according to any one of claims 1 to 4, characterized in that: The ontology knowledge information includes the attribute information of the element itself and the interaction information between the elements; the generation of the interaction information between the virtual and real elements based on the ontology knowledge information corresponding to the real scene and the virtual object respectively includes: According to the element's own attribute information and the interaction information between the elements contained in the ontology knowledge information, as well as the interaction rules, the interaction calculation between the virtual object and the real scene is performed to obtain the interaction information between the virtual and real elements; wherein the interaction rules include prior knowledge characterizing the interaction between the virtual and real elements.

6. A virtual-real fusion visual effect display system, characterized in that: include: Spatial ontology knowledge base, used to generate ontology knowledge information corresponding to real scenes and virtual objects; A scene interaction module is used to generate interaction information between virtual and real elements according to interaction rules; An interaction information generation module, used for mapping the interaction information between the virtual and real elements to a real scene video, so as to generate a fused scene video containing the interaction between the virtual and real elements; A fusion information analysis module, used to compare the fused scene video and the real scene video, and identify difference information; An adjustment module, used for dividing the visible area according to the difference information, and determining a transparency adjustment strategy corresponding to the video mask adjustment layer according to the division result; The video fusion module is used to execute the transparency adjustment strategy to insert a video mask adjustment layer between the fused scene video and the real scene video, and adjust the transparency of the video mask adjustment layer accordingly to show the interaction effect between the virtual object and the real scene.

7. The system according to claim 6, characterized in that Also includes: A scene information acquisition module, used to acquire the information of the real scene in real time, and generate a real scene video in real time based on the acquired real scene information; The virtual item information acquisition module is used to acquire the information of the virtual item.

8. A virtual-real fusion visual effect display device, characterized in that: include: A generation module configured to generate interaction information between virtual and real elements based on the ontological knowledge information corresponding to the real scene and the virtual object; An interaction module, configured to map the interaction information between the virtual and real elements to a real scene video to obtain a fused scene video containing the interaction of the virtual and real elements, wherein the real scene video is generated based on a real scene collected in real time; A comparison module, configured to compare the fused scene video with the real scene video to obtain difference information between the fused scene video and the real scene video; A fusion module is configured to insert a video mask adjustment layer between the fused scene video and the real scene video, and adjust the transparency of the video mask adjustment layer based on the difference information to display the interaction effect between the virtual object and the real scene.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images