A multi-person AR interaction method, device, equipment and storage medium

Abstract

The present invention discloses a multi-person AR interaction method, device, equipment and storage medium, the method comprising: acquiring real-time real scene data of a real space collected; constructing real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data comprises a real-time visual positioning map and a real-time three-dimensional interactive model; and merging the real-time virtual scene data with pre-acquired virtual content in real time to obtain a fused scene in the same scene. The embodiment of the present invention can realize real-time AR interaction among multiple people in the same scene, thereby improving user experience.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a multi-person AR interaction method, device, equipment and storage medium. Background Art

[0002] At present, most augmented reality (AR) experiences are single-player, lacking a continuous, shared experience. Even if multiple users play the same AR game and face the same scene, the game screens of multiple users are in different dimensions. Therefore, it is very necessary to study a multiplayer AR interaction method.

[0003] However, in the existing technology, most multi-person AR interaction methods cannot achieve interaction among multiple people in the same scene, resulting in a poor user experience. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-person AR interaction method, device, equipment and storage medium to solve the problem that the multi-person AR interaction method in the prior art cannot achieve interaction of multiple people in the same scene. The present invention can achieve real-time AR interaction of multiple people in the same scene by simultaneously constructing a real-time visual positioning map and a real-time three-dimensional interaction model, thereby improving the user experience.

[0005] To achieve the above object, an embodiment of the present invention provides a multi-person AR interaction method, comprising:

[0006] Obtain the real-time real scene data of the collected real space;

[0007] Constructing real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data includes a real-time visual positioning map and a real-time three-dimensional interactive model;

[0008] The real-time virtual scene data and the pre-acquired virtual content are fused in real time to obtain a fused scene in the same scene.

[0009] As an improvement to the above solution, the real-time visual positioning map is constructed by the following steps:

[0010] Extracting key frames of the real scene data;

[0011] Creating a sparse visual point cloud using the keyframes;

[0012] The real-time visual positioning map is constructed according to the sparse visual point cloud.

[0013] As an improvement to the above solution, the real-time three-dimensional interactive model is constructed by the following steps:

[0014] Establishing a dense point cloud based on the sparse visual point cloud;

[0015] The real-time three-dimensional interactive model is constructed according to the dense point cloud.

[0016] As an improvement of the above solution, the step of constructing the real-time three-dimensional interactive model according to the dense point cloud includes:

[0017] The dense point clouds are connected in series into geometric patches to obtain the real-time three-dimensional interactive model.

[0018] As an improvement to the above solution, the real-time three-dimensional interactive model is constructed by the following steps:

[0019] Acquire a real-time visual depth map of the real scene data;

[0020] The real-time three-dimensional interactive model is constructed according to the real-time visual depth map.

[0021] As an improvement to the above solution, a visual camera is used to collect the real-time real scene data.

[0022] To achieve the above object, an embodiment of the present invention further provides a multi-person AR interaction device, comprising:

[0023] A real-time real scene data acquisition module is used to acquire the collected real-time real scene data of the real space;

[0024] A real-time virtual scene data construction module, used to construct real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data includes a real-time visual positioning map and a real-time three-dimensional interactive model;

[0025] The virtual reality interaction module is used to fuse the real-time virtual scene data with the pre-acquired virtual content in real time to obtain a fused scene in the same scene.

[0026] To achieve the above objectives, an embodiment of the present invention further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the multi-person AR interaction method as described above when executing the computer program.

[0027] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the multi-person AR interaction method as described above.

[0028] Compared with the prior art, the multi-person AR interaction method, device, equipment and storage medium provided by the embodiments of the present invention enable each user to be located in the same scene and perform virtual reality interaction in the scene by constructing a real-time visual positioning map. By using the real-time real scene data of the real space to simultaneously construct a real-time visual positioning map and a real-time three-dimensional interaction model, the correct occlusion relationship and collision relationship between virtual and real objects can be achieved, thereby constructing a more natural, real, and highly immersive fusion scene, improving the user experience. Moreover, the embodiments of the present invention can continuously supplement and expand the virtual scene data by acquiring real-time real scene data, thereby expanding the fusion scene in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of a multi-person AR interaction method provided by an embodiment of the present invention;

[0030] Figure 2 is a structural block diagram of a multi-person AR interaction device provided by an embodiment of the present invention;

[0031] Figure 3 It is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1 , Figure 1 : is a flowchart of a multi-person AR interaction method provided by an embodiment of the present invention, the multi-person AR interaction method comprising:

[0034] S1, obtaining the real-time real scene data of the real space collected;

[0035] S2. constructing real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data includes a real-time visual positioning map and a real-time three-dimensional interactive model;

[0036] S3: Fusing the real-time virtual scene data and the pre-acquired virtual content in real time to obtain a fused scene in the same scene.

[0037] The embodiments of the present invention utilize a real-time visual positioning map to enable multiple users to be positioned in real time in their respective AR devices, so that each user is simultaneously located in the same physical scene, i.e., the same dimension, and can collide with and occlude objects in the physical scene at the same time.

[0038] The AR interaction method provided by the embodiment of the present invention is an incremental, crowdsourcing AR interaction method. As the interaction proceeds, multi-person AR devices can continuously obtain real-time real scene data on the basis of the original fusion scene, continuously supplement and expand virtual scene data, connect and repair the newly added three-dimensional interaction model with the geometric faces of the original three-dimensional interaction model to expand the entire three-dimensional interaction model, and continuously use the expanded three-dimensional interaction model data for fusion, so that the fusion scene is gradually expanded and complete, and a smooth experience of short-time preparation and incremental supplementation can be achieved.

[0039] Specifically, in step S1, the real-time real scene data is collected using a visual camera.

[0040] It is worth noting that the real scene data collected by the visual camera in real time is sent to the server to obtain real-time real scene data.

[0041] In an embodiment of the present invention, a visual camera is used to collect data of all people and objects in the real space in real time, and the data is used to build a real-time three-dimensional interaction model, so that correct occlusion relationships and collision relationships can be achieved during the interaction process.

[0042] In an optional embodiment, the real-time visual positioning map is constructed by the following steps:

[0043] Extracting key frames of the real scene data;

[0044] Creating a sparse visual point cloud using the keyframes;

[0045] The real-time visual positioning map is constructed according to the sparse visual point cloud.

[0046] In yet another optional embodiment, the real-time three-dimensional interactive model is constructed by the following steps:

[0047] Establishing a dense point cloud based on the sparse visual point cloud;

[0048] The real-time three-dimensional interactive model is constructed according to the dense point cloud.

[0049] Specifically, constructing the real-time three-dimensional interactive model according to the dense point cloud includes:

[0050] The dense point clouds are connected in series into geometric patches to obtain the real-time three-dimensional interactive model.

[0051] It is understandable that in the prior art, in order to realize the occlusion relationship during the interaction process, the 2D depth map collected by the sensor is usually used directly, but this method will result in incorrect occlusion relationships and collision relationships. For example, no matter how the real object moves, the virtual object always obscures the real object, which gives the user the illusion that the relative position between the virtual and real objects is distorted, resulting in a poor user experience. The present invention constructs a three-dimensional interaction model through three-dimensional reconstruction technology, which can achieve correct occlusion relationships and collision relationships during the interaction process. For example, according to the relative position between the virtual and real objects, the virtual object can be hidden behind the real object. Moreover, the embodiments of the present invention do not need to rely on active depth sensors, such as LIDAR, TOF, and structured light, but only need to use a visual camera to collect real scene data to construct a three-dimensional interaction model.

[0052] The embodiment of the present invention does not need to use an active depth sensor to obtain a visual depth map, but instead uses a three-dimensional reconstruction technology to connect the dense point cloud into geometric patches to obtain a real-time three-dimensional interactive model.

[0053] In yet another optional embodiment, the real-time three-dimensional interactive model is constructed by the following steps:

[0054] Acquire a real-time visual depth map of the real scene data;

[0055] The real-time three-dimensional interactive model is constructed according to the real-time visual depth map.

[0056] It is worth noting that the embodiments of the present invention may also utilize active depth sensors, such as LIDAR, TOF, and structured light, to obtain real-time visual depth maps.

[0057] Specifically, after step S3, the fused scene is sent to each AR device in real time, so that virtual reality interaction of each user in the same scene can be achieved.

[0058] The multi-person AR interaction method provided by the embodiment of the present invention enables each user to be located in the same scene and perform virtual reality interaction in the scene by constructing a real-time visual positioning map, and simultaneously constructs a real-time visual positioning map and a real-time three-dimensional interaction model by using the real-time real scene data of the real space, so as to achieve the correct occlusion relationship and correct collision relationship between virtual and real objects, thereby constructing a more natural, real, and highly immersive fusion scene, and improving the user experience. Moreover, the embodiment of the present invention can continuously supplement and expand the virtual scene data by acquiring real-time real scene data, and then expand the fusion scene in real time.

[0059] See also Figure 2 , Figure 2 : is a structural block diagram of a multi-person AR interactive device 10 provided in an embodiment of the present invention, wherein the multi-person AR interactive device 10 comprises:

[0060] A real-time real scene data acquisition module 11 is used to acquire the collected real-time real scene data of the real space;

[0061] A real-time virtual scene data construction module 12, used to construct real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data includes a real-time visual positioning map and a real-time three-dimensional interactive model;

[0062] The virtual reality interaction module 13 is used to fuse the real-time virtual scene data and the pre-acquired virtual content in real time to obtain a fused scene in the same scene.

[0063] Preferably, the real scene data acquisition module 11 includes:

[0064] The real scene data acquisition unit is used to acquire the real-time real scene data using a visual camera.

[0065] Preferably, the real-time virtual scene data construction module 12 includes:

[0066] A key frame extraction unit, used to extract the key frame of the real scene data;

[0067] A sparse visual point cloud building unit, used to build a sparse visual point cloud using the key frame;

[0068] A real-time visual positioning map construction unit is used to construct the real-time visual positioning map according to the sparse visual point cloud.

[0069] Preferably, the real-time virtual scene data construction module 12 further includes:

[0070] A dense point cloud building unit, used to build a dense point cloud according to the sparse visual point cloud;

[0071] The first real-time three-dimensional interactive model construction unit is used to construct the real-time three-dimensional interactive model according to the dense point cloud.

[0072] Preferably, the real-time three-dimensional interactive model construction unit is specifically used to connect the dense point clouds into geometric patches to obtain the real-time three-dimensional interactive model.

[0073] Preferably, the real-time virtual scene data construction module 12 further includes:

[0074] A real-time visual depth map acquisition unit, used to acquire a real-time visual depth map of the real scene data;

[0075] The second real-time three-dimensional interaction model construction unit is used to construct the real-time three-dimensional interaction model according to the real-time visual depth map.

[0076] It is worth noting that the working process of each module in the multi-person AR interaction device 10 described in the embodiment of the present invention can refer to the working process of the multi-person AR interaction method described in the above embodiment, and will not be repeated here.

[0077] A multi-person AR interactive device 10 provided in an embodiment of the present invention, by constructing a real-time visual positioning map, enables each user to be positioned in the same scene and perform virtual reality interaction in the scene. By using the real-time real scene data of the real space to simultaneously construct a real-time visual positioning map and a real-time three-dimensional interaction model, it is possible to achieve the correct occlusion relationship and correct collision relationship between virtual and real objects, thereby constructing a more natural, real, and highly immersive fusion scene, thereby improving the user experience. Moreover, by acquiring real-time real scene data, the embodiment of the present invention can continuously supplement and expand virtual scene data, thereby expanding the fusion scene in real time.

[0078] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the multi-person AR interaction method as described in the above embodiment.

[0079] See also Figure 3 , Figure 3 : is a structural block diagram of an electronic device 20 provided in an embodiment of the present invention, and the electronic device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned multi-person AR interaction method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0080] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device 20.

[0081] The electronic device 20 may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The electronic device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 20 and does not constitute a limitation on the electronic device 20. The electronic device 20 may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the electronic device 20 may also include input and output devices, network access devices, buses, etc.

[0082] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the electronic device 20, and uses various interfaces and lines to connect various parts of the entire electronic device 20.

[0083] The memory 22 can be used to store the computer program and / or module. The processor 21 realizes various functions of the electronic device 20 by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0084] Wherein, if the module / unit integrated in the electronic device 20 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 21. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0085] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0086] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-person AR interaction method, characterized in that: include: Obtain the real-time real scene data of the collected real space; Constructing real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data includes a real-time visual positioning map and a real-time three-dimensional interactive model constructed by three-dimensional reconstruction technology; The real-time virtual scene data and the pre-acquired virtual content are integrated in real time to obtain a fused scene in the same scene; The real-time visual positioning map is constructed by the following steps: Extracting key frames of the real scene data; Creating a sparse visual point cloud using the keyframes; Constructing the real-time visual positioning map according to the sparse visual point cloud; The real-time three-dimensional interactive model is constructed by the following steps: Establishing a dense point cloud based on the sparse visual point cloud; Constructing the real-time three-dimensional interactive model according to the dense point cloud; constructing the real-time three-dimensional interactive model according to the dense point cloud comprises: connecting the dense point cloud into geometric facets to obtain the real-time three-dimensional interactive model; The multi-person AR interaction method further comprises constructing the real-time three-dimensional interaction model by the following steps: Acquire a real-time visual depth map of the real scene data; The real-time three-dimensional interactive model is constructed according to the real-time visual depth map.

2. The multi-person AR interaction method according to claim 1, characterized in that: The real-time real scene data is collected using a visual camera.

3. A multi-person AR interactive device, characterized in that: include: A real-time real scene data acquisition module is used to acquire the collected real-time real scene data of the real space; A real-time virtual scene data construction module, used to construct real-time virtual scene data according to the real-time real scene data, wherein the real-time virtual scene data includes a real-time visual positioning map and a real-time three-dimensional interactive model constructed by three-dimensional reconstruction technology; A virtual reality interaction module, used for fusing the real-time virtual scene data with the pre-acquired virtual content in real time to obtain a fused scene in the same scene; Wherein, the real-time virtual scene data construction module includes: A key frame extraction unit, used to extract the key frame of the real scene data; A sparse visual point cloud building unit, used to build a sparse visual point cloud using the key frame; A real-time visual positioning map construction unit, used to construct the real-time visual positioning map according to the sparse visual point cloud; The real-time virtual scene data construction module also includes: A dense point cloud building unit, used to build a dense point cloud according to the sparse visual point cloud; A first real-time three-dimensional interactive model construction unit is used to construct the real-time three-dimensional interactive model according to the dense point cloud; the first real-time three-dimensional interactive model construction unit is specifically used to connect the dense point cloud into geometric facets to obtain the real-time three-dimensional interactive model; The real-time virtual scene data construction module also includes: A real-time visual depth map acquisition unit, used to acquire a real-time visual depth map of the real scene data; The second real-time three-dimensional interaction model construction unit is used to construct the real-time three-dimensional interaction model according to the real-time visual depth map.

4. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the multi-person AR interaction method according to any one of claims 1 to 2 when executing the computer program.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the multi-person AR interaction method according to any one of claims 1 to 2.

Images