A virtual-reality fusion method, device and system
Through the collaborative work of electronic devices and servers, and the use of 3D maps and environmental information, the seamless integration of the digital world and the physical world is achieved, solving the problem of insufficient positioning accuracy in existing AR technology, and improving rendering quality and resource utilization efficiency.
Patent Information
- Application Number
- CN202010225385.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-03-26
AI Technical Summary
The existing augmented reality (AR) technology cannot achieve the seamless integration of the digital world and the physical world, resulting in insufficient positioning accuracy and high sense of reality that is difficult to achieve.
Through the collaborative work of electronic devices and servers, the information collection module, spatial positioning module, scene understanding module, location determination module and rendering module are used, combined with 3D map and environmental information, the absolute position determination of electronic devices and the reasonable rendering of digital content is achieved, ensuring that The correct placement and display of digital content in the physical world.
It realizes the seamless integration of the digital world and the physical world, improves positioning accuracy and rendering quality, reduces the power and resource consumption of electronic devices, and ensures the privacy and high fidelity of digital content.
Smart Images

Figure CN113450439B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of augmented reality (AR) technology, and in particular to a virtual-reality fusion method, device and system. Background Art
[0002] AR technology is a technology that cleverly integrates virtual information with the real world. It widely uses multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing and other technical means to simulate computer-generated text, images, three-dimensional models, music, video and other virtual information and apply them to the real world. The two types of information complement each other, thus achieving "enhancement" of the real world. With the continuous development of computer vision technology, the application scope of AR technology is becoming wider and wider, but the existing AR technology cannot achieve seamless integration of the digital world and the physical world. Summary of the invention
[0003] The embodiment of the present invention discloses a virtual-reality fusion method, device and system for realizing seamless fusion of the digital world and the physical world.
[0004] In a first aspect, a virtual-reality fusion system is disclosed. The system may include an electronic device and a server. The electronic device may include an information acquisition module, a first spatial positioning module, a scene understanding module, a position determination module, a rendering module and a display module. The server may include a second spatial positioning module. The information acquisition module is used to acquire environmental information; the first spatial positioning module is used to send the environmental information to the server; the second spatial positioning module is used to determine the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and a stored 3D map, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device; the scene understanding module is used to determine the absolute position information and the 3D map corresponding to the absolute position information according to the environmental information and a stored 3D map. The environment information and the 3D map corresponding to the absolute position information identify the environment in which the electronic device is located; the position determination module is used to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result; the rendering module is used to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information; the display module is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0005] The virtual-reality fusion system disclosed in the embodiment of the present invention can obtain information about the current scene through the scene understanding module in the electronic device, and can calculate the reasonable placement position of the digital content in the entire 3D map, so as to render the 3D digital content at a suitable position, thereby realizing the seamless fusion of the digital world and the physical world. In addition, positioning through the server can reduce the power consumption required by the electronic device. At the same time, since the server can locate all electronic devices, it is not necessary for each electronic device to construct and store the 3D map separately, which can reduce the consumption of electronic device resources, thereby saving the resources of the electronic device.
[0006] As a possible implementation, the first spatial positioning module is also used to determine the environmental state of the electronic device according to the environmental information collection device, and send the environmental state to the server, wherein the environmental state includes an outdoor environment or an indoor environment; the second spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including: determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information, the environmental state and the stored 3D map.
[0007] The virtual-reality fusion system disclosed in the embodiment of the present invention can determine whether the environment of the electronic device is an outdoor environment or an indoor environment through an environmental information collection device, and thus adopt different 3D map search ranges for positioning, thereby improving positioning precision and accuracy.
[0008] As a possible implementation, the first spatial positioning module determines the environmental state of the electronic device according to the environmental information collection device, including: when the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment; when the environmental information collection device includes a second preset device, determining that the environmental state of the electronic device is an outdoor environment.
[0009] As a possible implementation, the environmental information may include an image, and the scene understanding module identifies the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information, including one or more of the following: identifying the scene in which the electronic device is located based on the image and the 3D map corresponding to the absolute position information; performing target detection on the image and the 3D map corresponding to the absolute position information; performing text detection on the image and the 3D map corresponding to the absolute position information; performing sign detection on the image and the 3D map corresponding to the absolute position information; performing material recognition based on the environmental information; performing lighting recognition based on the environmental information; performing scene depth calculation based on the image and the 3D map corresponding to the absolute position information; performing occlusion detection on the image and the 3D map corresponding to the absolute position information; performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; performing plane estimation on the image and the 3D map corresponding to the absolute position information; performing face recognition on the image; performing emotion recognition on the image; and performing target tracking based on the image and the 3D map corresponding to the absolute position information.
[0010] The virtual-reality fusion system disclosed in the embodiment of the present invention can obtain information such as the target, foreground, background, and spatial structure contained in the current scene through the scene understanding module, and can calculate the reasonable placement and mutual occlusion relationship of digital content in the real world, thereby realizing the seamless fusion of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, lighting estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the digital content from being incorrectly superimposed on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0011] As a possible implementation, the server may further include a user generated content (UGC) module, and the electronic device may further include a content acquisition module; the UGC module is used to generate the digital content; and the content acquisition module is used to acquire the digital content from the server.
[0012] In the virtual-reality fusion system disclosed in the embodiment of the present invention, digital content can be shared in the server. In addition, the privacy of the digital content can also be achieved through privacy settings.
[0013] As a possible implementation, the electronic device may further include a first encoding and decoding module, and the server may further include a second encoding and decoding module; the first encoding and decoding module is used to encode the environmental information; the first spatial positioning module sends the environmental information to the server, including: sending the encoded environmental information to the server; the second encoding and decoding module is used to decode the encoded environmental information to obtain the environmental information.
[0014] The virtual-reality fusion system disclosed in the embodiment of the present invention transmits environmental information that is encoded information collected by various sensors, thereby improving the reliability of information transmission.
[0015] As a possible implementation, the second encoding and decoding module is also used to encode the 3D map corresponding to the absolute position information; the second spatial positioning module sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, including: sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic device; the first encoding and decoding module is also used to decode the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0016] In the virtual-reality fusion system disclosed in the embodiment of the present invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, thereby improving the reliability of information transmission.
[0017] As a possible implementation, the second encoding and decoding module is also used to encode the digital content; the content acquisition module obtains the digital content from the server, including: obtaining the encoded digital content from the server; the first encoding and decoding module is also used to decode the encoded digital content to obtain the digital content.
[0018] In the virtual-reality fusion system disclosed in the embodiment of the present invention, the digital content transmitted is encoded digital content, so the reliability of information transmission can be improved.
[0019] As a possible implementation, the server may further include a training module; the training module is used to train a model based on training data and send the trained model to the electronic device; the scene understanding module identifies the environment in which the electronic device is located based on the 3D map corresponding to the environmental information and the absolute position information, including: identifying the environment in which the electronic device is located based on the trained model, the environmental information and the 3D map corresponding to the absolute position information.
[0020] The virtual-reality fusion system disclosed in the embodiment of the present invention can realize model sharing by having a server train a model used by an electronic device. In addition, it can also reduce the power consumption and resources required for training the model by the electronic device.
[0021] As a possible implementation, the server further includes a modeling module; the modeling module is used to obtain modeling data and construct a 3D map according to the modeling data, wherein the modeling data includes images and / or videos.
[0022] The virtual-reality fusion system disclosed in the embodiment of the present invention constructs a 3D map by a server, which can realize the sharing of 3D maps. In addition, it can also reduce the power consumption and resources required for the 3D map of electronic devices.
[0023] In a second aspect, an electronic device is disclosed, comprising an information collection module, a spatial positioning module, a scene understanding module, a position determination module, a rendering module and a display module, wherein: the information collection module is used to collect environmental information; the spatial positioning module is used to send the environmental information to the server, receive the absolute position information of the electronic device and the 3D map corresponding to the absolute position information from the server, the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map; the scene understanding module is used to identify the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information; the position determination module is used to determine the position of digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result; the rendering module is used to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information; the display module is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0024] The electronic device disclosed in the embodiment of the present invention can obtain information about the current scene through the scene understanding module in the electronic device, and can calculate the reasonable placement position of the digital content in the entire 3D map, so as to render the 3D digital content at a suitable position, thereby realizing the seamless integration of the digital world and the physical world. In addition, positioning through the server can reduce the power consumption required by the electronic device. At the same time, since the server can locate all electronic devices, it is not necessary for each electronic device to construct and store the 3D map separately, which can reduce the consumption of electronic device resources, thereby saving the resources of the electronic device.
[0025] As a possible implementation, the spatial positioning module is also used to determine the environmental state of the electronic device according to the environmental information collection device, and send the environmental state to the server, where the environmental state includes an outdoor environment or an indoor environment; the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map, including: the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information, the environmental state and the stored 3D map.
[0026] The electronic device disclosed in the embodiment of the present invention can determine whether the environment in which the electronic device is located is an outdoor environment or an indoor environment through an environmental information collection device, so as to adopt different 3D map search ranges for positioning, thereby improving positioning precision and accuracy.
[0027] As a possible implementation, the spatial positioning module determines the environmental state of the electronic device according to the environmental information collection device, including: when the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment; when the environmental information collection device includes a second preset device, determining that the environmental state of the electronic device is an outdoor environment.
[0028] As a possible implementation, the environmental information includes an image, and the scene understanding module identifies the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, including one or more of the following: identifying the scene in which the electronic device is located according to the image and the 3D map corresponding to the absolute position information; performing target detection on the image and the 3D map corresponding to the absolute position information; performing text detection on the image and the 3D map corresponding to the absolute position information; performing sign detection on the image and the 3D map corresponding to the absolute position information; performing material recognition according to the environmental information; performing lighting recognition according to the environmental information; performing scene depth calculation according to the image and the 3D map corresponding to the absolute position information; performing occlusion detection on the image and the 3D map corresponding to the absolute position information; performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; performing plane estimation on the image and the 3D map corresponding to the absolute position information; performing face recognition on the image; performing emotion recognition on the image; and performing target tracking according to the image and the 3D map corresponding to the absolute position information.
[0029] The electronic device disclosed in the embodiment of the present invention can obtain information such as the target, foreground, background, spatial structure, etc. contained in the current scene through the scene understanding module, and can calculate the reasonable placement position and mutual occlusion relationship of digital content in the real world, thereby realizing the seamless integration of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, lighting estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the mistaken superposition of digital content on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0030] As a possible implementation manner, the electronic device further includes a content acquisition module, wherein: the content acquisition module is used to acquire the digital content from the server.
[0031] In the electronic device disclosed in the embodiment of the present invention, digital content can be shared in the server, and the privacy of the digital content can also be achieved through privacy settings.
[0032] As a possible implementation, the electronic device further includes a coding module, wherein: the coding module is used to encode the environmental information; and the spatial positioning module sending the environmental information to the server includes: sending the encoded environmental information to the server.
[0033] The electronic device disclosed in the embodiment of the present invention transmits environmental information which is information collected by various sensors after being encoded, thus improving the reliability of information transmission.
[0034] As a possible implementation, the spatial positioning module receives the absolute position information of the electronic device from the server and the 3D map corresponding to the absolute position information, including: receiving the absolute position information of the electronic device from the server and the 3D map corresponding to the encoded absolute position information; the encoding and decoding module is also used to decode the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0035] In the electronic device disclosed in the embodiment of the present invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, and therefore, the reliability of information transmission can be improved.
[0036] As a possible implementation manner, the content acquisition module acquires the digital content from the server, including: acquiring encoded digital content from the server; and the encoding and decoding module is further configured to decode the encoded digital content to obtain the digital content.
[0037] In the electronic device disclosed in the embodiment of the present invention, the digital content transmitted is encoded digital content, so the reliability of information transmission can be improved.
[0038] A third aspect discloses a server, comprising a spatial positioning module, wherein: the spatial positioning module is used to receive environmental information from an electronic device, determine the absolute position information of the electronic device and a 3D map corresponding to the absolute position information based on the environmental information and a stored 3D map, send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information, determine the position of digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information based on the absolute position information and the recognition result, render the digital content based on the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and display the rendered digital content based on the position of the digital content in the 3D map corresponding to the absolute position information.
[0039] The server disclosed in the embodiment of the present invention can locate the electronic device through the environmental information of the electronic device, obtain the information of the current scene through the scene understanding module in the electronic device, and calculate the reasonable placement position of the digital content in the entire 3D map, so as to render the 3D digital content in the appropriate position, thereby realizing the seamless integration of the digital world and the physical world. In addition, positioning through the server can reduce the power consumption required by the electronic device. At the same time, since the server can locate all electronic devices, it is not necessary for each electronic device to construct and store the 3D map separately, which can reduce the consumption of electronic device resources, thereby saving the resources of the electronic device.
[0040] As a possible implementation, the spatial positioning module is also used to receive an environmental state from the electronic device, wherein the environmental state includes an outdoor environment or an indoor environment; the spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information based on the environmental information and the stored 3D map, including: determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information based on the environmental information, the environmental state and the stored 3D map.
[0041] The server disclosed in the embodiment of the present invention can determine whether the environment in which the electronic device is located is an outdoor environment or an indoor environment according to the environmental status, so as to adopt different 3D map search ranges for positioning, thereby improving the positioning precision and accuracy.
[0042] As a possible implementation, the environmental information includes an image, and the electronic device identifies the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, including one or more of the following: identifying the scene in which the electronic device is located according to the image and the 3D map corresponding to the absolute position information; performing target detection on the image and the 3D map corresponding to the absolute position information; performing text detection on the image and the 3D map corresponding to the absolute position information; performing sign detection on the image and the 3D map corresponding to the absolute position information; performing material recognition according to the environmental information; performing lighting recognition according to the environmental information; performing scene depth calculation according to the image and the 3D map corresponding to the absolute position information; performing occlusion detection on the image and the 3D map corresponding to the absolute position information; performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; performing plane estimation on the image and the 3D map corresponding to the absolute position information; performing face recognition on the image; performing emotion recognition on the image; and performing target tracking according to the image and the 3D map corresponding to the absolute position information.
[0043] The server disclosed in the embodiment of the present invention can obtain information such as the target, foreground, background, spatial structure, etc. contained in the current scene through the scene understanding module, and can calculate the reasonable placement and mutual occlusion relationship of digital content in the real world, thereby realizing the seamless integration of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, lighting estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the digital content from being incorrectly superimposed on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0044] As a possible implementation manner, the server further includes a user generated content (UGC) module, wherein: the UGC module is used to generate the digital content and send the digital content to the electronic device.
[0045] In the server disclosed in the embodiment of the present invention, digital content can be shared in the server. In addition, the privacy of the digital content can also be achieved through privacy settings.
[0046] As a possible implementation, the server further includes a coding and decoding module, wherein: the spatial positioning module receives environmental information from the electronic device, including: receiving encoded environmental information from the electronic device; the coding and decoding module is used to decode the encoded environmental information to obtain the environmental information.
[0047] The server disclosed in the embodiment of the present invention transmits environmental information which is encoded information collected by various sensors, thus improving the reliability of information transmission.
[0048] As a possible implementation, the encoding and decoding module is also used to encode the 3D map corresponding to the absolute position information; the spatial positioning module sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, including: sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic device.
[0049] The server disclosed in the embodiment of the present invention transmits a 3D map corresponding to the absolute position information, which is a 3D map corresponding to the encoded absolute position information. Therefore, the reliability of information transmission can be improved.
[0050] As a possible implementation manner, the encoding and decoding module is further used to encode the digital content; and the UGC module sending the digital content to the electronic device includes: sending the encoded digital content to the electronic device.
[0051] The digital content transmitted by the server disclosed in the embodiment of the present invention is encoded digital content, so the reliability of information transmission can be improved.
[0052] As a possible implementation, the server also includes a training module, wherein: the training module is used to train a model based on training data and send the trained model to the electronic device; the electronic device identifies the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information, including: identifying the environment in which the electronic device is located based on the trained model, the environmental information and the 3D map corresponding to the absolute position information.
[0053] The server disclosed in the embodiment of the present invention can train the model used by the electronic device, thereby realizing model sharing. In addition, it can also reduce the power consumption and resources required for the electronic device to train the model.
[0054] As a possible implementation, the server further includes a modeling module, wherein: the modeling module is used to obtain modeling data, and construct a 3D map according to the modeling data, and the modeling data includes images and / or videos.
[0055] The server disclosed in the embodiment of the present invention constructs a 3D map by the server, which can realize the sharing of the 3D map. In addition, it can also reduce the power consumption and resources required for the 3D map of the electronic device.
[0056] A fourth aspect discloses a virtual-reality fusion method, wherein an electronic device collects environmental information and sends the environmental information to a server; the server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information based on the environmental information and a stored 3D map, and sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device; the electronic device identifies the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information, determines the position of digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information based on the absolute position information and the recognition result, renders the digital content based on the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and displays the rendered digital content based on the position of the digital content in the 3D map corresponding to the absolute position information.
[0057] The virtual-reality fusion method disclosed in the embodiment of the present invention can obtain information about the current scene by identifying the environment in which the electronic device is located, and can calculate the reasonable placement position of the digital content in the entire 3D map, so as to render the 3D digital content at a suitable position, thereby realizing the seamless fusion of the digital world and the physical world. In addition, positioning through the server can reduce the power consumption required by the electronic device. At the same time, since the server can locate all electronic devices, it is not necessary for each electronic device to construct and store the 3D map separately, which can reduce the consumption of electronic device resources, thereby saving the resources of the electronic device.
[0058] As a possible implementation, the method also includes: the electronic device determines the environmental state of the electronic device according to the environmental information collection device, and sends the environmental state to the server, the environmental state includes an outdoor environment or an indoor environment; the server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including: the server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information, the environmental state and the stored 3D map.
[0059] The virtual-reality fusion method disclosed in the embodiment of the present invention can determine whether the environment of the electronic device is an outdoor environment or an indoor environment through an environmental information collection device, so as to adopt different 3D map search ranges for positioning, thereby improving positioning precision and accuracy.
[0060] As a possible implementation, the electronic device determines the environmental state of the electronic device based on the environmental information collection device, including: when the environmental information collection device includes a first preset device, the electronic device determines that the environmental state of the electronic device is an indoor environment; when the environmental information collection device includes a second preset device, the electronic device determines that the environmental state of the electronic device is an outdoor environment.
[0061] As a possible implementation manner, the environmental information includes an image, and the electronic device recognizes the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, including one or more of the following: the electronic device recognizes the scene in which the electronic device is located according to the image and the 3D map corresponding to the absolute position information; the electronic device performs target detection on the image and the 3D map corresponding to the absolute position information; the electronic device performs text detection on the image and the 3D map corresponding to the absolute position information; the electronic device performs sign detection on the image and the 3D map corresponding to the absolute position information; Material recognition; the electronic device performs lighting recognition based on the environmental information; the electronic device performs scene depth calculation based on the image and the 3D map corresponding to the absolute position information; the electronic device performs occlusion detection on the image and the 3D map corresponding to the absolute position information; the electronic device performs motion foreground detection on the image and the 3D map corresponding to the absolute position information; the electronic device performs plane estimation on the image and the 3D map corresponding to the absolute position information; the electronic device performs face recognition on the image; the electronic device performs emotion recognition on the image; the electronic device performs target tracking based on the image and the 3D map corresponding to the absolute position information.
[0062] The virtual-real fusion method disclosed in the embodiment of the present invention can obtain information such as the target, foreground, background, and spatial structure contained in the current scene, and can calculate the reasonable placement and mutual occlusion relationship of digital content in the real world, thereby realizing the seamless fusion of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, lighting estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the mistaken superposition of digital content on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0063] As a possible implementation manner, the method further includes: the server generating the digital content; and the electronic device acquiring the digital content from the server.
[0064] According to the virtual-reality integration method disclosed in the embodiment of the present invention, digital content can be shared in the server. In addition, the privacy of the digital content can also be achieved through privacy settings.
[0065] As a possible implementation, the method further includes: the electronic device encoding the environmental information; the electronic device sending the environmental information to the server includes: the electronic device sending the encoded environmental information to the server; the method further includes: the server decoding the encoded environmental information to obtain the environmental information.
[0066] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the environmental information transmitted is the information collected by various sensors after being encoded, and therefore, the reliability of information transmission can be improved.
[0067] As a possible implementation, the method also includes: the server encodes a 3D map corresponding to the absolute position information; the server sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, including: the server sends the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic device; the method also includes: the electronic device decodes the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0068] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, thereby improving the reliability of information transmission.
[0069] As a possible implementation, the method further includes: the server encoding the digital content; the electronic device obtaining the digital content from the server includes: the electronic device obtaining the encoded digital content from the server; the method further includes: the electronic device decoding the encoded digital content to obtain the digital content.
[0070] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the transmitted digital content is encoded digital content, thus improving the reliability of information transmission.
[0071] As a possible implementation, the method also includes: the server trains a model based on training data, and sends the trained model to the electronic device; the electronic device identifies the environment in which the electronic device is located based on a 3D map corresponding to the environmental information and the absolute position information, including: the electronic device identifies the environment in which the electronic device is located based on the trained model, the environmental information and the 3D map corresponding to the absolute position information.
[0072] The virtual-reality fusion method disclosed in the embodiment of the present invention can realize model sharing by having a server train a model used by an electronic device. In addition, it can also reduce the power consumption and resources required for training the model of the electronic device.
[0073] As a possible implementation, the method further includes: the server acquiring modeling data, and constructing a 3D map according to the modeling data, wherein the modeling data includes images and / or videos.
[0074] The virtual-reality fusion method disclosed in the embodiment of the present invention constructs a 3D map by a server, which can realize the sharing of 3D maps. In addition, it can also reduce the power consumption and resources required for the 3D map of electronic devices.
[0075] A fifth aspect discloses a virtual-reality fusion method, comprising: collecting environmental information; sending the environmental information to the server, receiving absolute position information of the electronic device and a 3D map corresponding to the absolute position information from the server, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map; identifying the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information; determining the position of digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result; rendering the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information; and displaying the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0076] The virtual-reality fusion method disclosed in the embodiment of the present invention can obtain information about the current scene by identifying the environment in which the electronic device is located, and can calculate the reasonable placement position of the digital content in the entire 3D map, so as to render the 3D digital content at a suitable position, thereby realizing the seamless fusion of the digital world and the physical world. In addition, positioning through the server can reduce the power consumption required by the electronic device. At the same time, since the server can locate all electronic devices, it is not necessary for each electronic device to construct and store the 3D map separately, which can reduce the consumption of electronic device resources, thereby saving the resources of the electronic device.
[0077] As a possible implementation, the method also includes: determining the environmental state of the electronic device according to the environmental information collection device, and sending the environmental state to the server, the environmental state including an outdoor environment or an indoor environment; the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map, including: the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information, the environmental state and the stored 3D map.
[0078] The virtual-reality fusion method disclosed in the embodiment of the present invention can determine whether the environment of the electronic device is an outdoor environment or an indoor environment through an environmental information collection device, so as to adopt different 3D map search ranges for positioning, thereby improving positioning precision and accuracy.
[0079] As a possible implementation, determining the environmental state of the electronic device based on the environmental information collection device includes: when the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment; when the environmental information collection device includes a second preset device, determining that the environmental state of the electronic device is an outdoor environment.
[0080] As a possible implementation, the environmental information includes an image, and the identifying the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information includes one or more of the following: identifying the scene in which the electronic device is located based on the image and the 3D map corresponding to the absolute position information; performing target detection on the image and the 3D map corresponding to the absolute position information; performing text detection on the image and the 3D map corresponding to the absolute position information; performing sign detection on the image and the 3D map corresponding to the absolute position information; performing material recognition based on the environmental information; performing lighting recognition based on the environmental information; performing scene depth calculation based on the image and the 3D map corresponding to the absolute position information; performing occlusion detection on the image and the 3D map corresponding to the absolute position information; performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; performing plane estimation on the image and the 3D map corresponding to the absolute position information; performing face recognition on the image; performing emotion recognition on the image; and performing target tracking based on the image and the 3D map corresponding to the absolute position information.
[0081] The virtual-real fusion method disclosed in the embodiment of the present invention can obtain information such as the target, foreground, background, and spatial structure contained in the current scene, and can calculate the reasonable placement and mutual occlusion relationship of digital content in the real world, thereby realizing the seamless fusion of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, lighting estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the mistaken superposition of digital content on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0082] As a possible implementation manner, the method further includes: acquiring the digital content from the server.
[0083] According to the virtual-reality integration method disclosed in the embodiment of the present invention, digital content can be shared in the server. In addition, the privacy of the digital content can also be achieved through privacy settings.
[0084] As a possible implementation manner, the method further includes: encoding the environmental information; and sending the environmental information to the server includes: sending the encoded environmental information to the server.
[0085] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the environmental information transmitted is the information collected by various sensors after being encoded, and therefore, the reliability of information transmission can be improved.
[0086] As a possible implementation, the receiving the absolute position information of the electronic device and the 3D map corresponding to the absolute position information from the server includes: receiving the absolute position information of the electronic device and the 3D map corresponding to the encoded absolute position information from the server; the method also includes: decoding the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0087] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, thereby improving the reliability of information transmission.
[0088] As a possible implementation manner, the obtaining the digital content from the server includes: obtaining encoded digital content from the server; and the method further includes: decoding the encoded digital content to obtain the digital content.
[0089] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the transmitted digital content is encoded digital content, thus improving the reliability of information transmission.
[0090] A sixth aspect discloses a virtual-reality fusion method, comprising: receiving environmental information from an electronic device; determining absolute position information of the electronic device and a 3D map corresponding to the absolute position information based on the environmental information and a stored 3D map; sending the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, wherein the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information, determining the position of digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information based on the absolute position information and the recognition result, rendering the digital content based on the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and displaying the rendered digital content based on the position of the digital content in the 3D map corresponding to the absolute position information.
[0091] The virtual-reality fusion method disclosed in the embodiment of the present invention locates the electronic device through the environmental information of the electronic device, obtains the information of the current scene by identifying the environment in which the electronic device is located, and calculates the reasonable placement position of the digital content in the entire 3D map, so as to render the 3D digital content in the appropriate position, thereby realizing the seamless fusion of the digital world and the physical world. In addition, positioning through the server can reduce the power consumption required by the electronic device. At the same time, since the server can locate all electronic devices, it is not necessary for each electronic device to construct and store the 3D map separately, which can reduce the consumption of electronic device resources, thereby saving the resources of the electronic device.
[0092] As a possible implementation, the method also includes: receiving an environmental state from the electronic device, the environmental state including an outdoor environment or an indoor environment; determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information based on the environmental information and the stored 3D map includes: determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information based on the environmental information, the environmental state and the stored 3D map.
[0093] The virtual-reality fusion method disclosed in the embodiment of the present invention can determine whether the environment in which the electronic device is located is an outdoor environment or an indoor environment through the environmental state, so as to adopt different 3D map search ranges for positioning, thereby improving the positioning precision and accuracy.
[0094] As a possible implementation, the environmental information includes an image, and the identifying the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information includes one or more of the following: identifying the scene in which the electronic device is located based on the image and the 3D map corresponding to the absolute position information; performing target detection on the image and the 3D map corresponding to the absolute position information; performing text detection on the image and the 3D map corresponding to the absolute position information; performing sign detection on the image and the 3D map corresponding to the absolute position information; performing material recognition based on the environmental information; performing lighting recognition based on the environmental information; performing scene depth calculation based on the image and the 3D map corresponding to the absolute position information; performing occlusion detection on the image and the 3D map corresponding to the absolute position information; performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; performing plane estimation on the image and the 3D map corresponding to the absolute position information; performing face recognition on the image; performing emotion recognition on the image; and performing target tracking based on the image and the 3D map corresponding to the absolute position information.
[0095] The virtual-real fusion method disclosed in the embodiment of the present invention can obtain information such as the target, foreground, background, and spatial structure contained in the current scene, and can calculate the reasonable placement and mutual occlusion relationship of digital content in the real world, thereby realizing the seamless fusion of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, lighting estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the mistaken superposition of digital content on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0096] As a possible implementation manner, the method further includes: generating the digital content; and sending the digital content to the electronic device.
[0097] According to the virtual-reality integration method disclosed in the embodiment of the present invention, digital content can be shared in the server. In addition, the privacy of the digital content can also be achieved through privacy settings.
[0098] As a possible implementation manner, the receiving of environmental information from the electronic device includes: receiving encoded environmental information from the electronic device; the method further includes: decoding the encoded environmental information to obtain the environmental information.
[0099] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the environmental information transmitted is the information collected by various sensors after being encoded, and therefore, the reliability of information transmission can be improved.
[0100] As a possible implementation, the method also includes: encoding a 3D map corresponding to the absolute position information; and sending the absolute position information and the 3D map corresponding to the absolute position information to the electronic device includes: sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic device.
[0101] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the 3D map corresponding to the transmitted absolute position information is the 3D map corresponding to the encoded absolute position information, thereby improving the reliability of information transmission.
[0102] As a possible implementation manner, the method further includes: encoding the digital content; and sending the digital content to the electronic device includes: sending the encoded digital content to the electronic device.
[0103] In the virtual-reality fusion method disclosed in the embodiment of the present invention, the transmitted digital content is encoded digital content, thus improving the reliability of information transmission.
[0104] As a possible implementation, the method also includes: training a model based on training data; sending the trained model to the electronic device; identifying the environment in which the electronic device is located based on the 3D map corresponding to the environmental information and the absolute position information includes: identifying the environment in which the electronic device is located based on the trained model, the environmental information and the 3D map corresponding to the absolute position information.
[0105] The virtual-reality fusion method disclosed in the embodiment of the present invention can realize model sharing by having a server train a model used by an electronic device. In addition, it can also reduce the power consumption and resources required for training the model of the electronic device.
[0106] As a possible implementation, the method further includes: acquiring modeling data, the modeling data including images and / or videos; and constructing a 3D map according to the modeling data.
[0107] The virtual-reality fusion method disclosed in the embodiment of the present invention constructs a 3D map by a server, which can realize the sharing of 3D maps. In addition, it can also reduce the power consumption and resources required for the 3D map of electronic devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0108] Figure 1 It is a structural schematic diagram of a virtual-reality fusion system disclosed in an embodiment of the present invention;
[0109] Figure 2 It is a structural schematic diagram of another virtual-reality fusion system disclosed in an embodiment of the present invention;
[0110] Figure 3is a schematic diagram of virtual-real fusion disclosed in an embodiment of the present invention;
[0111] Figure 4 is a schematic diagram of a virtual-reality fusion 3D navigation disclosed in an embodiment of the present invention;
[0112] Figure 5 is a structural schematic diagram of an electronic device disclosed in an embodiment of the present invention;
[0113] Figure 6 is a schematic diagram of the structure of another electronic device disclosed in an embodiment of the present invention;
[0114] Figure 7 It is a structural schematic diagram of a server disclosed in an embodiment of the present invention;
[0115] Figure 8 It is a flowchart of a virtual-reality fusion method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION
[0116] The embodiments of the present invention disclose a virtual-reality fusion method, device and system for realizing seamless fusion of the digital world and the physical world, which are described in detail below.
[0117] In order to better understand the virtual-real fusion method, device and system disclosed in the embodiment of the present invention, the application scenario of the embodiment of the present invention is described below. AR technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. After simulating computer-generated text, images, three-dimensional models, music, videos and other virtual information, they are applied to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world. With the continuous development of electronic technology and computer vision technology, some relatively mature AR applications have gradually appeared on electronic devices, such as the ARKit platform launched by Apple and the ARCore platform launched by Google. These AR applications can estimate the position of electronic devices, the posture of electronic devices, plane estimation, and illumination estimation. The above-mentioned AR applications have limited understanding of the 3D physical world, and a full understanding of the physical world is an important basis for realizing highly realistic AR technology. Only by identifying the structure, semantics, three-dimensional (3D) information, illumination, and material of the physical world can the seamless nesting of AR content be achieved.
[0118] At present, a digital grid model of a city can be built based on high-resolution two-dimensional (2D) satellite images of ground data to generate map features. When an electronic device is enabled to shoot, the system matches the visual information with the map features. Therefore, it is able to determine the absolute position of the electronic device. However, the above situation only supports outdoor positioning of large scenes and does not have indoor positioning capabilities. In addition, since the technical solution is limited to satellite images, the outdoor positioning accuracy is insufficient, and the error reaches more than 3m, which cannot achieve a highly realistic fusion of the physical world and the digital world.
[0119] It can be seen that AR technology can only provide limited AR functions and cannot achieve the seamless integration of the digital world and the physical world.
[0120] See also Figure 1 , Figure 1 Schematic diagram of a virtual-reality fusion system disclosed in an embodiment of the present invention. Figure 1 As shown, the virtual-reality fusion system may include an electronic device 11 and a server 12. The electronic device 11 may include an information acquisition module 111, a first spatial positioning module 112, a scene understanding module 113, a position determination module 114, a rendering module 115, and a display module 116. The server 12 may include a second spatial positioning module 121, wherein:
[0121] An information collection module 111 is used to collect environmental information;
[0122] A first spatial positioning module 112, used to send environmental information to the server 12;
[0123] The second spatial positioning module 121 is used to determine the absolute position information of the electronic device 11 and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 11;
[0124] A scene understanding module 113, configured to identify the environment in which the electronic device 11 is located based on the 3D map corresponding to the environmental information and the absolute position information;
[0125] A position determination module 114, configured to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device 11 in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result;
[0126] A rendering module 115, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device 11 in the 3D map corresponding to the absolute position information;
[0127] The display module 116 is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0128] When the user uses the virtual-reality fusion system, the electronic device 11 can collect environmental information through the information collection module 111. Specifically, the electronic device 11 can collect environmental information through one or more sensors such as a camera, wireless fidelity (WiFi), Bluetooth, global positioning system (GPS), barometer, geomagnetic sensor, inertial measurement unit (IMU), 5G, and light sensor. The camera is used to collect images and / or videos, the WiFi sensor is used to collect Wifi fingerprint signals, the Bluetooth sensor is used to collect Bluetooth signals, the GPS is used to collect geographic location information, the barometer is used to measure atmospheric pressure, the geomagnetic sensor is used to detect the earth's magnetism, the IMU is used to measure the three-axis attitude angle and acceleration of the electronic device 11, the light sensor is used to collect light, and 5G is used to collect 5G base station signals. Environmental information may include images, light, multi-spectrum, depth, material, geographic location information, etc. Multi-spectrum, depth, and material can be collected and calculated by multi-spectrum sensors.
[0129] Afterwards, the electronic device 11 can send the collected environmental information to the server 12 through the first spatial positioning module 112. Wireless communication or wired communication can be used between the electronic device 11 and the server 12. The server 12 can be a cloud server, an edge server, or other types of servers. After the server 12 receives the environmental information from the electronic device 11, it can determine the absolute position information of the electronic device 11 and the 3D map corresponding to the absolute position information through the second spatial positioning module 121 according to the environmental information and the stored 3D map. The absolute position information of the electronic device 11 can be determined using a pre-trained positioning model, and the environmental information and the stored 3D map can be input into the positioning model. The output of the positioning model is the absolute position information of the electronic device 11. Afterwards, the 3D map corresponding to the absolute position information can be determined according to the absolute position information and the stored 3D map, that is, the 3D map corresponding to the absolute geographical location information is obtained from the 3D map stored in the server 12, that is, the 3D map where the current position of the electronic device 11 is located is obtained from the stored 3D map. Afterwards, the server 12 can send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 11. The 3D map stored in the server 12 is a 3D map pre-built by the server 12. The absolute position information is the position information in the earth coordinate system.
[0130] After the electronic device 11 receives the absolute position information and the 3D map corresponding to the absolute position information from the server 12, the scene understanding module 113 can identify the environment in which the electronic device 11 is located according to the environment information combined with the 3D map corresponding to the absolute position information. The recognition result can include information such as the target, foreground, background, and spatial structure information contained in the current scene. The scene in which the electronic device 11 is located can be identified, and the scene recognition can be performed by a pre-trained scene recognition model. The recognition can be performed based on the image included in the environment information combined with the 3D map corresponding to the absolute position information. The recognition result can be a specific geographical location, the name of the scene, or a certain same scene in the database. Object detection (i.e., target detection) can also be performed on the image included in the environment information combined with the 3D map corresponding to the absolute position information, that is, objects in the image included in the environment information and the 3D map corresponding to the absolute position information are detected, and the detection can be performed through a trained object detection model. Text in the environment in which the electronic device 11 is located can also be detected, and the text appearing in the image included in the environment information and the 3D map corresponding to the absolute position information can be detected by a pre-trained text detection model. When performing text detection, optical character recognition (OCR) can be used, or other text recognition models can be used. It is also possible to detect logos in the environment where the electronic device 11 is located, and the image included in the environmental information can be detected by a pre-trained logo detection model in combination with the 3D map corresponding to the absolute position information. It is also possible to identify the material of the object in the environment where the electronic device 11 is located, that is, to identify the material of the object in the environmental information. It is also possible to identify the lighting in the environment where the electronic device 11 is located, that is, to identify the lighting in the environmental information, and the lighting can include the intensity of the lighting, the brightness of the light, etc. It is also possible to calculate the scene depth, which is the distance between the content seen by the camera and the camera, and can be measured or calculated using time of flight ranging (TOF), structured light or software algorithms. It is also possible to detect occlusion in the environment where the electronic device 11 is located, and the image included in the environmental information can be detected by a pre-trained occlusion detection model in combination with the 3D map corresponding to the absolute position information. It is also possible to detect the motion foreground in the environment where the electronic device 11 is located, and the image included in the environmental information can be detected by a pre-trained motion foreground detection model in combination with a 3D map corresponding to the absolute position information. It is also possible to perform a plane estimation on the environment where the electronic device 11 is located. It is also possible to recognize faces in the environment where the electronic device 11 is located, and the faces in the images included in the environmental information can be recognized and annotated by a pre-trained face recognition model.It is also possible to identify the emotions of the faces in the environment where the electronic device 11 is located, and to identify and annotate the emotions of the faces in the images included in the environmental information through a pre-trained emotion recognition model. It is also possible to track some information in the environment where the electronic device 11 is located, and to track and annotate some information based on the images included in the environmental information through a pre-trained tracking model, and the information may be scenes, logos, faces, etc. The images included in the environmental information may be images captured by a camera, or images included in a video captured by a camera.
[0131] Afterwards, the position determination module 114 can determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device 11 in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result of the scene understanding module 113. The rendering module 115 can render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result of the scene understanding module 113 and the position of the electronic device 11 in the 3D map corresponding to the absolute position information. The display module 116 can display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information. The scene understanding module 113 can obtain information such as the target, foreground, background, and spatial structure contained in the current scene, and can calculate the reasonable placement position and mutual occlusion relationship of the digital content in the real world, thereby realizing the seamless splicing of the digital world and the physical world. At the same time, high-fidelity 3D rendering based on global illumination technology can be realized with the help of 3D graphics rendering engine, illumination estimation, material estimation and digital content geometric structure. The foreground and background analysis technology of scene understanding can avoid the digital content from being incorrectly superimposed on the foreground, and the motion information of scene understanding can also ensure the continuity of the rendering effect.
[0132] See also Figure 2 , Figure 2 is a schematic diagram of the structure of another virtual-real fusion system disclosed in an embodiment of the present invention. Figure 2 Is Figure 1 Optimized. Figure 2 As shown, the virtual-reality fusion system may include an electronic device 21 and a server 22, the electronic device 21 may include an information acquisition module 211, a first spatial positioning module 212, a scene understanding module 213, a position determination module 214, a rendering module 215 and a display module 216, and the server 22 may include a second spatial positioning module 221, wherein:
[0133] An information collection module 211 is used to collect environmental information;
[0134] A first spatial positioning module 212, used to send environmental information to the server 22;
[0135] The second spatial positioning module 221 is used to determine the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 21;
[0136] A scene understanding module 213, used to identify the environment in which the electronic device 21 is located based on the 3D map corresponding to the environmental information and the absolute position information;
[0137] A position determination module 214, configured to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device 21 in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result;
[0138] A rendering module 215, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device 21 in the 3D map corresponding to the absolute position information;
[0139] The display module 216 is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0140] A detailed description can be found in Figure 1 The corresponding description is not repeated here.
[0141] In one embodiment, the information collection module 211 may collect environmental information by:
[0142] Environmental information is collected through one or more of cameras, WiFi, Bluetooth, GPS, barometer, IMU, 5G, and geomagnetic sensors.
[0143] A detailed description can be found in Figure 1 The corresponding descriptions are not repeated here.
[0144] In another embodiment, the first spatial positioning module 212 is further used to determine the environmental state of the electronic device 21 according to the environmental information collection device, and send the environmental state to the server 22. The environmental state may include an outdoor environment or an indoor environment.
[0145] The second spatial positioning module 221 determines the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including:
[0146] The absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information are determined according to the environmental information, the environmental state and the stored 3D map.
[0147] In yet another embodiment, the first spatial positioning module 212 determines the environmental state of the electronic device 21 according to the environmental information collection device, including:
[0148] In the case where the environmental information collection device includes the first preset device, determining that the environmental state of the electronic device 21 is an indoor environment;
[0149] In the case that the environmental information collecting device includes the second preset device, it is determined that the environmental state of the electronic device 21 is an outdoor environment.
[0150] The environmental state of the electronic device 21 can be an indoor environment or an outdoor environment. The 3D map stored in the server 22 may include an indoor 3D map and an outdoor 3D map. Therefore, when positioning the electronic device 21, the environmental state of the electronic device 21 can be determined first. The electronic device 21 can determine the environmental state of the electronic device 21 according to the environmental information collection device through the first spatial positioning module 212. When the environmental information collection device includes a first preset device, it can be determined that the environmental state of the electronic device 21 is an indoor environment. The first preset device may include WiFi, Bluetooth, etc., and the first preset device may also include 5G, a camera, etc. When the environmental information collection device includes a second preset device, it can be determined that the environmental state of the electronic device 21 is an outdoor environment. The second preset device may include GPS, etc., and the second preset device may also include 5G, a camera, etc. After that, the electronic device 21 sends the environmental state to the server 22 through the first spatial positioning module 212. After the server 22 receives the environmental status from the electronic device 21, it can determine the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the environmental information, the environmental status and the stored 3D map, that is, determine the absolute position information of the electronic device 21 and the 3D map corresponding to the absolute position information according to the 3D map corresponding to the environmental information and the environmental status. The second spatial positioning module 221 in the server 22 can include two parts, one part for outdoor positioning and the other part for indoor positioning. Therefore, after the server 22 receives the environmental information and environmental status from the electronic device 21, it can first determine whether to select outdoor positioning or indoor positioning according to the environmental status, and then perform positioning according to the indoor or outdoor map and the environmental information. It can be seen that positioning can include outdoor positioning and indoor positioning, and the positioning accuracy of outdoor positioning and indoor positioning can be different. Generally speaking, since the indoor space is small, the positioning error is required to be small, that is, the positioning accuracy is high. However, the outdoor positioning space is large, and the positioning error can be slightly larger, that is, the positioning accuracy is small. The absolute position information of the electronic device 21 changes as the electronic device 21 moves. During the movement of the electronic device 21, the absolute position information of the electronic device 21 can be determined in real time through a position tracking algorithm, such as a visual inertial odometry (VIO).
[0151] In another embodiment, the environmental information may include an image, and the scene understanding module 213 identifies the environment in which the electronic device 21 is located according to the environmental information and the 3D map corresponding to the absolute position information, including one or more of the following:
[0152] Identify the scene where the electronic device 21 is located according to the 3D map corresponding to the image and absolute position information included in the environmental information;
[0153] Performing target detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0154] Perform text detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0155] Performing logo detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0156] Identify materials based on environmental information;
[0157] Perform lighting recognition based on environmental information;
[0158] Calculate scene depth based on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0159] Perform occlusion detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0160] Perform motion foreground detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0161] Performing plane estimation on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0162] Performing face recognition on images included in environmental information;
[0163] Perform emotion recognition on images including environmental information;
[0164] Target tracking is performed based on the 3D map corresponding to the image and absolute position information included in the environmental information.
[0165] Among them, the detailed description of the above content can refer to the relevant description of the scene understanding module 113 above, which will not be repeated here.
[0166] After the rendering module 216 renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device 21 in the 3D map corresponding to the absolute position information, the display module 116 can display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, and can display the rendered digital content at the position of the digital content in the 3D map corresponding to the absolute position information.
[0167] In yet another embodiment, the server 22 may further include a user generated content (UGC) module 222, and the electronic device 21 may further include a content acquisition module 217, wherein:
[0168] UGC module 222, for generating digital content;
[0169] The content acquisition module 217 is used to acquire the digital content from the server 22 .
[0170] The digital content is generated by the server, and then the electronic device 21 can obtain it from the server 22 through the content acquisition module 217, which can be obtained together with the absolute position information, or can be obtained at other times, which is not limited here. The content acquisition module can send an acquisition request for obtaining digital content to the server 22. After receiving the acquisition request from the electronic device 21, the server 22 can send the digital content generated by the UGC module 222 to the electronic device 21. In addition, the acquisition request can carry an identifier, and the server can send the digital content identified by the identifier to the electronic device. Digital content can be shared in the server, and the privacy of digital content can also be achieved through privacy settings.
[0171] Users can customize digital content as personalized content in the digital world. A human-computer interface and content creation tools can be provided to facilitate users to create their own digital content and place it in the UGC module 222. By setting a user privacy mechanism, users can publish personalized content in the entire digital world, or set it to be visible in a small range as content input for rendering.
[0172] In yet another embodiment, the electronic device 21 may further include a first codec module 218, and the server 22 may further include a second codec module 223, wherein:
[0173] A first encoding and decoding module 218, used to encode environmental information;
[0174] The first spatial positioning module 212 sends environmental information to the server 22 including:
[0175] Sending the encoded environment information to the server 22;
[0176] The second encoding and decoding module 223 is used to decode the encoded environment information to obtain the environment information.
[0177] The first encoding and decoding module 218 encodes the environmental information, which may be all of the environmental information or part of the environmental information, such as images, videos, etc.
[0178] In yet another embodiment, the second encoding and decoding module 223 is further used to encode the 3D map corresponding to the absolute position information;
[0179] The second spatial positioning module 221 sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device 21, including:
[0180] Sending the absolute position information and the 3D map corresponding to the encoded absolute position information to the electronic device 21;
[0181] The first encoding and decoding module 218 is further used to decode the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0182] In yet another embodiment, the second encoding and decoding module 223 is further used to encode the digital content;
[0183] The content acquisition module 217 acquires the digital content from the server 22 including:
[0184] Obtaining the encoded digital content from the server 22;
[0185] The first encoding and decoding module 218 is further used to decode the encoded digital content to obtain the digital content.
[0186] In yet another embodiment, the server 22 may further include a training module 224, wherein:
[0187] A training module 224, used for training a model according to training data and sending the trained model to the electronic device 21;
[0188] The scene understanding module 213 identifies the environment in which the electronic device 21 is located according to the 3D map corresponding to the environmental information and the absolute position information, including:
[0189] The environment where the electronic device 21 is located is identified based on the trained model, the environmental information and the 3D map corresponding to the absolute position information.
[0190] The scene understanding module 213 of the electronic device 21 uses a plurality of trained models, which can be trained by the server 22 through the training module 224 according to the training data. The training data can be data obtained by the server 22 from other electronic devices. The training data can be images, videos, etc.
[0191] In yet another embodiment, the server 22 may further include a modeling module 225, wherein:
[0192] The modeling module 225 is used to obtain modeling data and construct a 3D map according to the modeling data. The modeling data may include images and / or videos.
[0193] The server 22 can obtain modeling data through the modeling module 225, and can build a 3D map based on the obtained modeling data. It mainly realizes the construction of indoor 3D maps and outdoor 3D maps. In addition, the modeling module 225 can use real-time collected images and videos to update the 3D map. According to the different characteristics of indoor 3D maps and outdoor 3D maps, different collection methods can be used to collect modeling data for building maps. For example, images and / or videos used to build outdoor 3D maps can be collected by means of drones, satellites, aerial photography, panorama, etc., while images and / or video maps used to build indoor 3D maps can be collected by means of laser scanning, panoramic cameras, etc. The construction of the 3D map can be carried out offline by the server 22, which can ensure its accuracy and effect. In addition, since the real-world 3D scenes often change over time, the 3D map should be continuously updated. The map update can be automatically updated with the help of images and videos collected by users, as well as the corresponding spatial location information and video tracking information, or it can be updated by the publisher of the 3D map by regularly supplementing data.
[0194] Among them, the 3D map is placed on the server and can be called by all users, which can also reduce the storage resources used by electronic devices. The UGC module saves the user's personalized digital content. The UGC module is set on the server, which is convenient for sharing and adopts appropriate life cycle and privacy settings. The scene understanding module is set on the electronic device to avoid uploading the image to the server, thereby protecting the user's privacy. The rendering module is set on the electronic device to avoid uploading the fusion of digital content of the current scene to the server, thereby leaking user privacy. In addition, the rendering module is set on the electronic device to improve the efficiency of virtual and real fusion, thereby improving the user experience.
[0195] For example, an electronic device collects video and geographic location information, and sends the video and geographic location information to a server. The video is collected by a camera, and the geographic location information can be collected by a visual positioning system (VPS), GPS, location based services (LBS), etc. The server determines the absolute location information of the electronic device based on the video, geographic location information and 3D map, and sends the absolute location information of the electronic device to the electronic device. The electronic device continuously obtains the relative position and posture of the camera through simultaneous localization and mapping (SLAM), and couples with the VPS to continuously obtain the absolute position and posture of the camera in the world coordinate system (that is, the absolute position information of the electronic device). The electronic device analyzes the scene understanding information of the real world, such as lighting information, material information, etc., and combines the relative position and posture of the camera to realistically render the digital content superimposed on the physical world, thereby achieving virtual-real occlusion, navigation, holographic signage and other effects. Displaying the rendered digital content can present the virtual-real fusion effect to the user. Please refer to Figure 3 , Figure 3 is a schematic diagram of virtual-real fusion disclosed in an embodiment of the present invention. Figure 3 As shown in Figure 1, digital content, such as convenience store signs and virtual large screens, is integrated with the physical world. Figure 4 , Figure 4 Schematic diagram of a virtual-real fusion 3D navigation disclosed in an embodiment of the present invention. Figure 4 As shown, the route is indicated by virtual digital content.
[0196] See also Figure 5 , Figure 5 Schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Figure 5 As shown, the electronic device may include an information acquisition module 51, a spatial positioning module 52, a scene understanding module 53, a position determination module 54, a rendering module 55 and a display module 56, wherein:
[0197] An information collection module 51 is used to collect environmental information;
[0198] The spatial positioning module 52 is used to send environmental information to the server, and receive absolute position information and a 3D map corresponding to the absolute position information from the server, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map;
[0199] A scene understanding module 53, used to identify the environment in which the electronic device is located based on the 3D map corresponding to the environmental information and the absolute position information;
[0200] A position determination module 54, configured to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result;
[0201] A rendering module 55, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information;
[0202] The display module 56 is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0203] A detailed description can be found in Figure 1 The corresponding description is not repeated here.
[0204] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the structure of another electronic device disclosed in an embodiment of the present invention. Figure 6 As shown, the electronic device may include an information acquisition module 61, a spatial positioning module 62, a scene understanding module 63, a position determination module 64, a rendering module 65 and a display module 66, wherein:
[0205] An information collection module 61 is used to collect environmental information;
[0206] A spatial positioning module 62, used to send environmental information to a server, and receive absolute position information and a 3D map corresponding to the absolute position information from the server, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map;
[0207] A scene understanding module 63, used to identify the environment in which the electronic device is located based on the 3D map corresponding to the environmental information and the absolute position information;
[0208] A position determination module 64, configured to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result;
[0209] A rendering module 65, configured to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information;
[0210] The display module 66 is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0211] A detailed description can be found in Figure 1 The corresponding description is not repeated here.
[0212] In another embodiment, the information collection module 61 may collect environmental information by:
[0213] Environmental information is collected through one or more of cameras, WiFi, Bluetooth, GPS, barometer, IMU, 5G, and geomagnetic sensors.
[0214] A detailed description can be found in Figure 1 The corresponding descriptions are not repeated here.
[0215] In yet another embodiment, the spatial positioning module 62 is further used to determine the environmental state of the electronic device according to the environmental information collection device, and send the environmental state to the server. The environmental state may include an outdoor environment or an indoor environment;
[0216] The absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map and include:
[0217] The absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information, the environment status and the stored 3D map.
[0218] In yet another embodiment, the spatial positioning module 62 determines the environmental state of the electronic device according to the environmental information collection device, including:
[0219] In a case where the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment;
[0220] In the case where the environmental information collecting device includes the second preset device, it is determined that the environmental state of the electronic device is an outdoor environment.
[0221] Among them, the specific description can refer to the corresponding description above, which will not be repeated here.
[0222] In yet another embodiment, the environmental information may include an image, and the scene understanding module 63 identifies the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, including one or more of the following:
[0223] Identify the scene where the electronic device is located according to the 3D map corresponding to the image and absolute position information included in the environmental information;
[0224] Performing target detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0225] Perform text detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0226] Performing logo detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0227] Identify materials based on environmental information;
[0228] Perform lighting recognition based on environmental information;
[0229] Calculate scene depth based on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0230] Perform occlusion detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0231] Perform motion foreground detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0232] Performing plane estimation on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0233] Performing face recognition on images included in environmental information;
[0234] Perform emotion recognition on images including environmental information;
[0235] Target tracking is performed based on the 3D map corresponding to the image and absolute position information included in the environmental information.
[0236] Among them, the detailed description of the above content can refer to the relevant description of the scene understanding module 113 above, which will not be repeated here.
[0237] In yet another embodiment, the electronic device may further include a content acquisition module 67, wherein:
[0238] The content acquisition module 67 is used to acquire digital content from a server.
[0239] Among them, the detailed description of the content acquisition module can refer to the corresponding description above, which will not be repeated here.
[0240] In yet another embodiment, the electronic device may further include a codec module 68, wherein:
[0241] The codec module 68 is used for environmental information;
[0242] The spatial positioning module 62 sends environmental information to the server including:
[0243] Sends the encoded environment information to the server.
[0244] In yet another embodiment, the spatial positioning module 62 receives the absolute position information of the electronic device from the server and the 3D map corresponding to the absolute position information, including:
[0245] Receiving the absolute position information of the electronic device and the 3D map corresponding to the encoded absolute position information from the server;
[0246] The encoding and decoding module 68 is further used to decode the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0247] In yet another embodiment, the content acquisition module 67 acquires digital content from the server including:
[0248] Obtain the encoded digital content from the server;
[0249] The encoding and decoding module 68 is also used to decode the encoded digital content to obtain digital content.
[0250] For a detailed description of each of the above modules of the electronic device, reference can be made to the above related descriptions and will not be repeated here.
[0251] See also Figure 7 , Figure 7 Schematic diagram of the structure of a server disclosed in an embodiment of the present invention. Figure 7 As shown, the server may include a spatial positioning module 71, wherein:
[0252] The spatial positioning module 71 is used to receive environmental information from the electronic device, determine the absolute position information of the electronic device and the 3D map corresponding to the absolute position information based on the environmental information and the stored 3D map, send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify the environment in which the electronic device is located based on the environmental information and the 3D map corresponding to the absolute position information, determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information based on the absolute position information and the recognition result, render the digital content based on the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and display the rendered digital content based on the position of the digital content in the 3D map corresponding to the absolute position information.
[0253] In another embodiment, the spatial positioning module 71 is further used to receive an environmental state from the electronic device, and the environmental state may include an outdoor environment or an indoor environment;
[0254] The spatial positioning module 71 determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including:
[0255] The absolute position information of the electronic device and the 3D map corresponding to the absolute position information are determined according to the environmental information, the environmental state and the stored 3D map.
[0256] In yet another embodiment, the environmental information may include an image, and the electronic device identifying the environment in which the electronic device is located according to the environmental information and a 3D map corresponding to the absolute position information includes one or more of the following:
[0257] Identify the scene where the electronic device is located based on the 3D map corresponding to the image and the absolute position information;
[0258] Perform object detection on the 3D map corresponding to the image and absolute position information;
[0259] Perform text detection on the image and the 3D map corresponding to the absolute position information;
[0260] Perform landmark detection on the 3D map corresponding to the image and absolute position information;
[0261] Identify materials based on environmental information;
[0262] Perform lighting recognition based on environmental information;
[0263] Calculate scene depth based on the 3D map corresponding to the image and absolute position information;
[0264] Perform occlusion detection on the 3D map corresponding to the image and absolute position information;
[0265] Perform motion foreground detection on the 3D map corresponding to the image and absolute position information;
[0266] Perform plane estimation on the 3D map corresponding to the image and absolute position information;
[0267] Perform face recognition on images;
[0268] Perform emotion recognition on images;
[0269] Target tracking is performed based on the 3D map corresponding to the image and absolute position information.
[0270] In yet another embodiment, the server may further include a UGC module 72, wherein:
[0271] The UGC module 72 is used to generate digital content and send the digital content to electronic devices.
[0272] In yet another embodiment, the server may further include a codec module 73, wherein:
[0273] The environment information received by the spatial positioning module 71 from the electronic device includes:
[0274] Receiving coded environmental information from electronic devices;
[0275] The encoding and decoding module 73 is used to decode the encoded environmental information to obtain the environmental information.
[0276] In yet another embodiment, the encoding and decoding module 73 is further used to encode the 3D map corresponding to the absolute position information;
[0277] The spatial positioning module 71 sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, including:
[0278] The absolute position information and the 3D map corresponding to the encoded absolute position information are sent to the electronic device.
[0279] In yet another embodiment, the codec module 73 is further used to encode the digital content;
[0280] The UGC module 72 sends digital content to the sub-device including:
[0281] Send the encoded digital content to the electronic device.
[0282] In yet another embodiment, the server may further include a training module 74, wherein:
[0283] A training module 74, used to train a model according to training data and send the trained model to an electronic device;
[0284] The electronic device identifies the environment in which the electronic device is located according to the 3D map corresponding to the environmental information and the absolute position information, including:
[0285] The environment in which the electronic device is located is identified based on the 3D map corresponding to the trained model, environmental information and absolute position information.
[0286] In yet another embodiment, the server may further include a modeling module 75, wherein:
[0287] The modeling module 75 is used to obtain modeling data and construct a 3D map according to the modeling data, where the modeling data includes images and / or videos.
[0288] For a detailed description of each module on the server, please refer to the above related description, which will not be repeated here.
[0289] See also Figure 8 , Figure 8 FIG. 1 is a flow chart of a virtual-real fusion method disclosed in an embodiment of the present invention. Figure 8As shown, the virtual-reality fusion method may include the following steps.
[0290] 801. Electronic devices collect environmental information.
[0291] Electronic devices can collect environmental information through one or more of cameras, WiFi, Bluetooth, GPS, barometer, IMU, 5G, and geomagnetic sensors.
[0292] 802. The electronic device sends environmental information to the server.
[0293] Accordingly, the server receives the environment information from the electronic device.
[0294] 803. The server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map.
[0295] 804. The server sends the absolute location information and the 3D map corresponding to the absolute location information to the electronic device.
[0296] 805. The electronic device identifies the environment in which the electronic device is located according to the 3D map corresponding to the environmental information and the absolute position information.
[0297] 806. The electronic device determines, based on the absolute position information and the recognition result, the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information.
[0298] 807 . The electronic device renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information.
[0299] 808. The electronic device displays the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
[0300] Optionally, the above method may further include:
[0301] The electronic device determines the environment state of the electronic device according to the environmental information collection device, and sends the environment state to the server, where the environment state may include an outdoor environment or an indoor environment;
[0302] Step 805 is specifically as follows:
[0303] The absolute position information of the electronic device and the 3D map corresponding to the absolute position information are determined according to the environmental information, the environmental state and the stored 3D map.
[0304] Optionally, the electronic device determines the environmental state of the electronic device according to the environmental information collection device, including:
[0305] In a case where the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment;
[0306] In the case where the environmental information collecting device includes the second preset device, it is determined that the environmental state of the electronic device is an outdoor environment.
[0307] Optionally, the environment information may include an image, and the electronic device identifying the environment in which the electronic device is located according to the environment information and a 3D map corresponding to the absolute position information includes one or more of the following:
[0308] The electronic device identifies the scene where the electronic device is located according to the 3D map corresponding to the image and absolute position information included in the environmental information;
[0309] The electronic device performs target detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0310] The electronic device performs text detection on the 3D map corresponding to the image and the absolute position information included in the environmental information;
[0311] The electronic device performs landmark detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0312] Electronic devices identify materials based on environmental information;
[0313] Electronic devices perform light recognition based on environmental information;
[0314] The electronic device calculates scene depth based on the image and the 3D map corresponding to the absolute position information included in the environmental information;
[0315] The electronic device performs occlusion detection on the 3D map corresponding to the image and the absolute position information included in the environmental information;
[0316] The electronic device performs motion foreground detection on the 3D map corresponding to the image and absolute position information included in the environmental information;
[0317] The electronic device performs plane estimation on the 3D map corresponding to the image and the absolute position information included in the environmental information;
[0318] The electronic device performs face recognition on the image included in the environmental information;
[0319] The electronic device performs emotion recognition on the image included in the environmental information;
[0320] The electronic device tracks the target according to the 3D map corresponding to the image and absolute position information included in the environmental information.
[0321] Optionally, the above method may further include:
[0322] The server generates digital content;
[0323] The electronic device obtains the digital content from the server.
[0324] Optionally, the above method may further include:
[0325] Electronic devices encode environmental information;
[0326] The electronic device sends the environmental information to the server including:
[0327] The electronic device sends the encoded environmental information to the server;
[0328] The above method may further include:
[0329] The server decodes the encoded environment information to obtain the environment information.
[0330] Optionally, the above method may further include:
[0331] The server encodes the 3D map corresponding to the absolute position information;
[0332] The server sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, including:
[0333] The server sends the absolute location information and the 3D map corresponding to the encoded absolute location information to the electronic device;
[0334] The above method may further include:
[0335] The electronic device decodes the 3D map corresponding to the encoded absolute position information to obtain the 3D map corresponding to the absolute position information.
[0336] Optionally, the above method may further include:
[0337] The server encodes the digital content;
[0338] The electronic device obtains the digital content from the server, including:
[0339] The electronic device obtains the encoded digital content from the server;
[0340] The above method may further include:
[0341] The electronic device decodes the encoded digital content to obtain digital content.
[0342] Optionally, the above method may further include:
[0343] The server trains the model according to the training data and sends the trained model to the electronic device;
[0344] The electronic device identifies the environment in which the electronic device is located according to the 3D map corresponding to the environmental information and the absolute position information, including:
[0345] The electronic device identifies the environment in which the electronic device is located based on the trained model, environmental information, and a 3D map corresponding to the absolute position information.
[0346] Optionally, the above method may further include:
[0347] The server obtains modeling data and constructs a 3D map according to the modeling data, where the modeling data includes images and / or videos.
[0348] The details of each step above can be found in the above description and will not be elaborated here.
[0349] The contents of the above embodiments can be referenced to each other. The content of each embodiment is not limited to this embodiment, but can also be applied to corresponding contents in other embodiments.
[0350] The specific implementation methods described above are further detailed descriptions of the objectives, technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.
Claims
1. A virtual-reality fusion system, characterized in that: The electronic device comprises an information acquisition module, a first spatial positioning module, a scene understanding module, a position determination module, a rendering module and a display module, and the server comprises a second spatial positioning module, wherein: The information collection module is used to collect environmental information; The first spatial positioning module is used to send the environmental information to the server; The second spatial positioning module is used to determine the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, and send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device; The scene understanding module is used to identify the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information; The position determination module is used to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result; The rendering module is used to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information; The display module is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
2. The system according to claim 1, characterized in that The first spatial positioning module is further used to determine the environmental state of the electronic device according to the environmental information collection device, and send the environmental state to the server, where the environmental state includes an outdoor environment or an indoor environment; The second spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including: The absolute position information of the electronic device and the 3D map corresponding to the absolute position information are determined according to the environmental information, the environmental state and the stored 3D map.
3. The system according to claim 2, characterized in that The first spatial positioning module determines the environmental state of the electronic device according to the environmental information collection device, including: In a case where the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment; In a case where the environmental information collecting device includes a second preset device, it is determined that the environmental state of the electronic device is an outdoor environment.
4. The system according to any one of claims 1 to 3, characterized in that: The environmental information includes an image, and the scene understanding module identifies the environment in which the electronic device is located according to the environmental information and a 3D map corresponding to the absolute position information, including one or more of the following: Identifying a scene where the electronic device is located according to the image and a 3D map corresponding to the absolute position information; Performing target detection on the image and the 3D map corresponding to the absolute position information; Performing text detection on the image and the 3D map corresponding to the absolute position information; Performing landmark detection on the image and the 3D map corresponding to the absolute position information; Perform material identification according to the environmental information; Performing lighting recognition according to the environmental information; Calculating scene depth based on the image and the 3D map corresponding to the absolute position information; Performing occlusion detection on the image and the 3D map corresponding to the absolute position information; Performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; Performing plane estimation on the image and the 3D map corresponding to the absolute position information; Performing face recognition on the image; performing emotion recognition on the image; The target is tracked according to the image and the 3D map corresponding to the absolute position information.
5. The system according to any one of claims 1 to 3, characterized in that: The server further includes a user generated content (UGC) module, and the electronic device further includes a content acquisition module, wherein: The UGC module is used to generate the digital content; The content acquisition module is used to acquire the digital content from the server.
6. The system according to any one of claims 1 to 3, characterized in that: The server also includes a training module, wherein: The training module is used to train the model according to the training data and send the trained model to the electronic device; The scene understanding module identifies the environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information, including: The environment in which the electronic device is located is identified according to the trained model, the environmental information and a 3D map corresponding to the absolute position information.
7. The system according to any one of claims 1 to 3, characterized in that: The server also includes a modeling module, wherein: The modeling module is used to obtain modeling data and construct a 3D map according to the modeling data, wherein the modeling data includes images and / or videos.
8. An electronic device, characterized in that: It includes information acquisition module, spatial positioning module, scene understanding module, position determination module, rendering module and display module, among which: The information collection module is used to collect environmental information; The spatial positioning module is used to send the environmental information to the server, and receive the absolute position information of the electronic device and the 3D map corresponding to the absolute position information from the server, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map; The scene understanding module is used to identify the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information; The position determination module is used to determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result; The rendering module is used to render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information; The display module is used to display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
9. The electronic device according to claim 8, characterized in that: The spatial positioning module is further used to determine the environmental state of the electronic device according to the environmental information collection device, and send the environmental state to the server, where the environmental state includes an outdoor environment or an indoor environment; The absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environment information and the stored 3D map, including: The absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information, the environmental state and the stored 3D map.
10. The electronic device according to claim 9, characterized in that: The spatial positioning module determines the environmental state of the electronic device according to the environmental information collection device, including: In a case where the environmental information collection device includes a first preset device, determining that the environmental state of the electronic device is an indoor environment; In a case where the environmental information collecting device includes a second preset device, it is determined that the environmental state of the electronic device is an outdoor environment.
11. The electronic device according to any one of claims 8 to 10, characterized in that: The environmental information includes an image, and the scene understanding module identifies the environment in which the electronic device is located according to the environmental information and a 3D map corresponding to the absolute position information, including one or more of the following: Identifying a scene where the electronic device is located according to the image and a 3D map corresponding to the absolute position information; Performing target detection on the image and the 3D map corresponding to the absolute position information; Performing text detection on the image and the 3D map corresponding to the absolute position information; Performing landmark detection on the image and the 3D map corresponding to the absolute position information; Perform material identification according to the environmental information; Performing lighting recognition according to the environmental information; Calculating scene depth based on the image and the 3D map corresponding to the absolute position information; Performing occlusion detection on the image and the 3D map corresponding to the absolute position information; Performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; Performing plane estimation on the image and the 3D map corresponding to the absolute position information; Performing face recognition on the image; performing emotion recognition on the image; The target is tracked according to the image and the 3D map corresponding to the absolute position information.
12. The electronic device according to any one of claims 8 to 10, characterized in that: The electronic device further comprises a content acquisition module, wherein: The content acquisition module is used to acquire the digital content from the server.
13. A server, characterized in that: Includes a spatial positioning module, including: The spatial positioning module is used to receive environmental information from the electronic device, determine the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, send the absolute position information and the 3D map corresponding to the absolute position information to the electronic device, the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, determine the position of digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result, render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
14. The server according to claim 13, characterized in that: The spatial positioning module is further used to receive an environmental state from the electronic device, where the environmental state includes an outdoor environment or an indoor environment; The spatial positioning module determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including: The absolute position information of the electronic device and the 3D map corresponding to the absolute position information are determined according to the environmental information, the environmental state and the stored 3D map.
15. The server according to claim 13 or 14, characterized in that: The server also includes a user generated content (UGC) module, wherein: The UGC module is used to generate the digital content and send the digital content to the electronic device.
16. The server according to claim 13 or 14, characterized in that: The server also includes a training module, wherein: The training module is used to train the model according to the training data and send the trained model to the electronic device; The electronic device identifying the environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information includes: The environment in which the electronic device is located is identified according to the trained model, the environmental information and a 3D map corresponding to the absolute position information.
17. The server according to claim 13 or 14, characterized in that: The server also includes a modeling module, wherein: The modeling module is used to obtain modeling data and construct a 3D map according to the modeling data, wherein the modeling data includes images and / or videos.
18. A virtual-real fusion method, characterized in that: include: The electronic device collects environmental information and sends the environmental information to the server; The server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, and sends the absolute position information and the 3D map corresponding to the absolute position information to the electronic device; The electronic device identifies the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, determines the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result, renders the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and displays the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
19. The method according to claim 18, characterized in that The method further comprises: The electronic device determines the environmental state of the electronic device according to the environmental information collection device, and sends the environmental state to the server, where the environmental state includes an outdoor environment or an indoor environment; The server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map, including: The server determines the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information, the environmental state and the stored 3D map.
20. The method according to claim 19, characterized in that The electronic device determining the environmental state of the electronic device according to the environmental information collection device includes: In a case where the environmental information collection device includes a first preset device, the electronic device determines that the environmental state of the electronic device is an indoor environment; In a case where the environmental information collecting device includes a second preset device, the electronic device determines that the environmental state of the electronic device is an outdoor environment.
21. The method according to any one of claims 18 to 20, characterized in that: The environmental information includes an image, and the electronic device identifies the environment in which the electronic device is located according to the environmental information and a 3D map corresponding to the absolute position information, including one or more of the following: The electronic device identifies a scene where the electronic device is located according to the image and a 3D map corresponding to the absolute position information; The electronic device performs target detection on the image and the 3D map corresponding to the absolute position information; The electronic device performs text detection on the image and the 3D map corresponding to the absolute position information; The electronic device performs landmark detection on the image and the 3D map corresponding to the absolute position information; The electronic device performs material identification according to the environmental information; The electronic device performs light recognition according to the environmental information; The electronic device calculates scene depth according to the image and the 3D map corresponding to the absolute position information; The electronic device performs occlusion detection on the image and the 3D map corresponding to the absolute position information; The electronic device performs motion foreground detection on the image and the 3D map corresponding to the absolute position information; The electronic device performs plane estimation on the image and the 3D map corresponding to the absolute position information; The electronic device performs face recognition on the image; The electronic device performs emotion recognition on the image; The electronic device tracks the target according to the image and the 3D map corresponding to the absolute position information.
22. The method according to any one of claims 18 to 20, characterized in that: The method further comprises: The server generates the digital content; The electronic device obtains the digital content from the server.
23. The method according to any one of claims 18 to 20, characterized in that: The method further comprises: The server trains a model according to the training data and sends the trained model to the electronic device; The electronic device identifying the environment in which the electronic device is located according to the environment information and the 3D map corresponding to the absolute position information includes: The electronic device identifies the environment in which the electronic device is located according to the trained model, the environmental information and a 3D map corresponding to the absolute position information.
24. The method according to any one of claims 18 to 20, characterized in that: The method further comprises: The server acquires modeling data and constructs a 3D map according to the modeling data, wherein the modeling data includes images and / or videos.
25. A virtual-real fusion method, characterized in that: include: Collect environmental information; Sending the environmental information to a server, receiving absolute position information of the electronic device and a 3D map corresponding to the absolute position information from the server, wherein the absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map; Identifying the environment in which the electronic device is located according to the environmental information and a 3D map corresponding to the absolute position information; Determine, according to the absolute position information and the recognition result, the position of the digital content in the 3D map corresponding to the absolute position information, and the position of the electronic device in the 3D map corresponding to the absolute position information; Rendering the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result, and the position of the electronic device in the 3D map corresponding to the absolute position information; The rendered digital content is displayed according to the position of the digital content in the 3D map corresponding to the absolute position information.
26. The method according to claim 25, characterized in that The method further comprises: Determine the environment state of the electronic device according to the environmental information collection device, and send the environment state to the server, where the environment state includes an outdoor environment or an indoor environment; The absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information and the stored 3D map and include The absolute position information and the 3D map corresponding to the absolute position information are determined by the server according to the environmental information, the environmental state and the stored 3D map.
27. The method according to claim 25 or 26, characterized in that The environmental information includes an image, and identifying the environment in which the electronic device is located according to the environmental information and a 3D map corresponding to the absolute position information includes one or more of the following: Identifying a scene where the electronic device is located according to the image and a 3D map corresponding to the absolute position information; Performing target detection on the image and the 3D map corresponding to the absolute position information; Performing text detection on the image and the 3D map corresponding to the absolute position information; Performing landmark detection on the image and the 3D map corresponding to the absolute position information; Perform material identification according to the environmental information; Performing lighting recognition according to the environmental information; Calculating scene depth based on the image and the 3D map corresponding to the absolute position information; Performing occlusion detection on the image and the 3D map corresponding to the absolute position information; Performing motion foreground detection on the image and the 3D map corresponding to the absolute position information; Performing plane estimation on the image and the 3D map corresponding to the absolute position information; Performing face recognition on the image; performing emotion recognition on the image; The target is tracked according to the image and the 3D map corresponding to the absolute position information.
28. A virtual-real fusion method, characterized in that: include: receiving environmental information from the electronic device; Determining the absolute position information of the electronic device and the 3D map corresponding to the absolute position information according to the environmental information and the stored 3D map; The absolute position information and a 3D map corresponding to the absolute position information are sent to the electronic device, wherein the absolute position information and the 3D map corresponding to the absolute position information are used to instruct the electronic device to identify the environment in which the electronic device is located according to the environmental information and the 3D map corresponding to the absolute position information, determine the position of the digital content in the 3D map corresponding to the absolute position information and the position of the electronic device in the 3D map corresponding to the absolute position information according to the absolute position information and the recognition result, render the digital content according to the position of the digital content in the 3D map corresponding to the absolute position information, the recognition result and the position of the electronic device in the 3D map corresponding to the absolute position information, and display the rendered digital content according to the position of the digital content in the 3D map corresponding to the absolute position information.
29. The method according to claim 28, characterized in that The method further comprises: Receiving an environmental state from the electronic device, the environmental state comprising an outdoor environment or an indoor environment; The determining, according to the environmental information and the stored 3D map, the absolute position information of the electronic device and the 3D map corresponding to the absolute position information comprises: The absolute position information of the electronic device and the 3D map corresponding to the absolute position information are determined according to the environmental information, the environmental state and the stored 3D map.
30. The method according to claim 28 or 29, characterized in that The method further comprises: generating the digital content; The digital content is transmitted to the electronic device.
