Method and device for sharing virtual vision based on visual map fusion

Abstract

The present application relates to the field of VR technology and provides a method and device for sharing virtual vision based on visual map fusion. A reference visual map is selected from the initial visual maps sent by multiple virtual terminals. For each key frame in any non-reference visual map, when there is a common viewing area between the key frame and the historical key frame in the reference visual map, the non-reference visual map and the reference visual map are fused by aligning the map coordinate systems of different virtual terminals to a single map coordinate system to obtain a target visual map. Based on the fused target visual map, the status information of virtual objects in the virtual vision shared by different virtual terminals is synchronized. After the target visual map and the status information are respectively sent to the corresponding virtual terminals, the different virtual terminals can accurately and efficiently share the virtual vision.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and in particular to a method and device for fusing and sharing virtual vision based on visual maps. Background Art

[0002] Shared virtual vision is a key technology for realizing multi-person interaction based on augmented reality (AR) / mixed reality (MR) terminals, which enables users to experience the fun of interacting in the virtual and real worlds.

[0003] Currently, there are two main methods for achieving virtual field of view sharing: in method one, the MultipeerConnectivity framework is used to transmit visual map data between an AR terminal and its nearby AR terminals. By matching the feature-rich visual map of the target area, the two successfully matched AR terminals are aligned in the coordinate system of the target area, thereby achieving virtual field of view sharing; in method two, Cloud Anchors technology is used to perform plane detection on the point cloud data collected by the AR terminal, create a sparse 3D feature point cloud of the target area as an anchor point, and upload the anchor point to the cloud server. After other AR terminals using Cloud Anchors technology obtain the anchor point of the same target area, they match the 3D feature point cloud with the existing anchor point in the cloud server, and align the coordinate system of the two successfully matched AR terminals in the target area, thereby achieving virtual field of view sharing.

[0004] However, Method 1 and Method 2 require taking multiple images around the target area to obtain rich visual information to improve the matching quality. The operation steps are relatively complicated and the user-friendliness is low. Moreover, Method 1 and Method 2 cannot solve the problem of misjudgment of similar areas, that is, the problem of incorrect recognition caused when different AR terminals are in two target areas that are visually similar but actually different. In addition, Method 1 is limited by the MultipeerConnectivity framework and can only align the coordinate systems between terminals that are relatively close. Terminals that are farther away cannot upload visual map data, and the functional scalability and data persistence are insufficient. Method 2 is limited to the anchor points when aligning the coordinate system, not the coordinate system alignment between the entire visual map. For virtual scenes that rely on the coordinate system itself rather than the anchor points, the virtual field of view sharing effect is poor. Summary of the Invention

[0005] The embodiments of the present application provide a method and device for sharing a virtual field of view based on visual map fusion, which is used to enable multiple virtual terminals to share a virtual field of view.

[0006] In one aspect, an embodiment of the present application provides a method for fusing and sharing a virtual field of view based on a visual map, comprising:

[0007] receiving initial visual maps sent by multiple virtual terminals, and determining a reference visual map from the multiple initial visual maps;

[0008] For each keyframe in any non-reference visual map, determining whether the keyframe has a common view area with a historical keyframe in a historical keyframe library, where the historical keyframe library is generated based on the keyframes included in the multiple initial visual maps;

[0009] When there is a common viewing area, determining whether the key frame and the historical key frame belong to different virtual terminals;

[0010] When belonging to different virtual terminals, performing map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame to obtain a target visual map;

[0011] The state information of the virtual objects in the common viewing area is synchronized according to the target visual map, and the state information and the target visual map are respectively sent to corresponding virtual terminals to realize the sharing of virtual vision.

[0012] On the other hand, an embodiment of the present application provides a server, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0013] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0014] receiving, via the communication interface, initial visual maps sent by a plurality of virtual terminals, and determining a reference visual map from the plurality of initial visual maps;

[0015] For each keyframe in any non-reference visual map, determining whether the keyframe has a common view area with a historical keyframe in a historical keyframe library, where the historical keyframe library is generated based on the keyframes included in the multiple initial visual maps;

[0016] When there is a common viewing area, determining whether the key frame and the historical key frame belong to different virtual terminals;

[0017] When belonging to different virtual terminals, performing map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame to obtain a target visual map;

[0018] The state information of the virtual objects in the common viewing area is synchronized according to the target visual map, and the state information and the target visual map are respectively sent to corresponding virtual terminals through the communication interface to realize the sharing of virtual vision.

[0019] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a method for fusing a shared virtual vision based on a visual map.

[0020] The beneficial effects of the embodiments of the present application are as follows:

[0021] Embodiments of the present application provide a method and device for sharing a virtual field of view based on visual map fusion. A historical keyframe library is generated based on keyframes included in initial visual maps sent by multiple virtual terminals, and a reference visual map is selected from the multiple initial visual maps. For each keyframe in any non-reference visual map, when the keyframe has a common viewing area with a historical keyframe in the historical keyframe library, it is determined whether the keyframe and the historical keyframe belong to different virtual terminals. A map fusion is performed on the non-reference visual map corresponding to the keyframe and the reference visual map corresponding to the historical keyframe to obtain a target visual map. The map coordinate systems of the multiple virtual terminals are unified to the same map coordinate system based on the map coordinate system of the selected reference visual map. This ensures that the coordinate system of each virtual terminal is consistent with the map coordinate system of the server, facilitating map fusion. Furthermore, based on the fused target visual map, status information of virtual objects in the virtual field of view shared by different virtual terminals is synchronized. After the target visual map and status information are sent to the corresponding virtual terminals, each virtual terminal can accurately and efficiently share the virtual field of view.

[0022] Other features and advantages of the embodiments of the present application will be described in the following description, and in part will become apparent from the description or be understood by practicing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1This is an effect diagram of multiple terminals in the same area not sharing a virtual field of view provided in an embodiment of the present application;

[0025] Figure 2 This is an effect diagram of the lack of shared virtual field of view due to misalignment of the coordinate systems of the terminal and the server provided in the embodiment of the present application;

[0026] Figure 3 This is a diagram of the overall architecture for implementing virtual vision sharing provided in an embodiment of the present application;

[0027] Figure 4 A flowchart of a method for generating a visual map according to an embodiment of the present application;

[0028] Figure 5 A flow chart of a method for sharing virtual vision based on visual map fusion provided in an embodiment of the present application;

[0029] Figure 6 A flow chart of a method for determining a common view area provided in an embodiment of the present application;

[0030] Figure 7 2D visual features extracted from the key frames of the non-reference visual map provided in the embodiment of the present application;

[0031] Figure 8 2D visual features extracted from historical key frames of the reference visual map provided in the embodiment of the present application;

[0032] Figure 9 Flowchart of the visual map fusion method provided in the embodiment of the present application;

[0033] Figure 10 Flowchart of a 3D point cloud matching method between a key frame and a historical key frame provided in an embodiment of the present application;

[0034] Figure 11 2D visual features extracted from the key frames provided in the embodiments of this application;

[0035] Figure 12 2D visual features extracted from historical key frames provided in the embodiments of this application;

[0036] Figure 13 A schematic diagram of the data structure for describing 3D points using BREIF features of 2D points provided in an embodiment of the present application;

[0037] Figure 14 The initial 3D point cloud corresponding to the key frames and historical key frames provided in the embodiments of the present application;

[0038] Figure 15 The target 3D point cloud corresponding to the key frames and historical key frames provided in the embodiments of the present application;

[0039] Figure 16 The motion trajectory diagram obtained by fusion of the two visual maps provided in the embodiment of the present application;

[0040] Figure 17 This is an effect diagram of two virtual terminals in the same area sharing a virtual field of view provided by an embodiment of the present application;

[0041] Figure 18 This is a diagram showing the effect of aligning the coordinate system of the virtual terminal and the server provided in an embodiment of the present application;

[0042] Figure 19 The embodiment of the present application provides two virtual terminals that see virtual objects with the same posture at the same location in the real world;

[0043] Figure 20 A structural diagram of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0045] All other embodiments derived by persons of ordinary skill in the art based on the exemplary embodiments described herein without inventive effort are within the scope of protection of the claims appended hereto. Furthermore, although the disclosure herein is presented based on one or more exemplary embodiments, it should be understood that each aspect of the disclosure may constitute a complete embodiment on its own.

[0046] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0047] exist Figure 1In the application scenario shown, AR / MR terminal 1 and AR / MR terminal 2 in the same area are connected to the cloud server respectively and send their respective visual maps to the cloud server. The cloud server establishes a map coordinate system based on a visual map. Assume that the map coordinate system is established using the visual map sent by AR / MR terminal 1. When two users in the same area observe the same virtual object generated by the cloud server based on their respective AR / MR terminals, because the coordinate system of AR / MR terminal 1 is aligned with the map coordinate system of the cloud server, while the coordinate system of AR / MR terminal 2 is not aligned with the map coordinate system of the cloud server, the two AR / MR terminals cannot share the virtual field of view. As a result, user A and user B will find that the same virtual object is located in different locations in the real world and has different postures. Therefore, the coordinate systems in the two AR / MR terminals need to be aligned.

[0048] Optionally, the cloud server may select a visual map based on the order of received visual maps to establish a map coordinate system, or may randomly select a visual map to establish a map coordinate system.

[0049] exist Figure 2 In the application scenario shown, after a visual map is pre-established based on an AR / MR terminal, the visual map is sent to the cloud server and stored. The cloud server establishes a map coordinate system based on the stored visual map. When another AR / MR terminal establishes a connection with the cloud server, when the user observes the virtual object generated by the cloud server in the area of ​​the visual map based on the AR / MR terminal he is wearing, the coordinate system of the newly connected AR / MR terminal is inconsistent with the map coordinate system of the cloud server. It will be found that the position (including position and posture) of the virtual object in the real world is different from the data returned by the cloud server. Therefore, it is necessary to fuse the visual map of the newly connected AR / MR terminal with the visual map stored on the cloud server.

[0050] It should be noted that Figure 1 and Figure 2 This is just an example, and there is no restriction on the number of AR / MR terminals.

[0051] The embodiments of the present application are based on Simultaneous Localization And Mapping (SLAM) technology, multi-point projection positioning technology, and 3D point cloud alignment technology, and provide a method and device for sharing a virtual field of view based on visual map fusion. This method can keep the coordinate system of the AR / MR terminal consistent with the map coordinate system of the server, and can efficiently and accurately fuse the visual maps of multiple AR / MR terminals, thereby realizing a method for AR / MR terminals to share a virtual field of view.

[0052] See also Figure 3, is the overall architecture diagram provided by the embodiment of the present application, in which data is transmitted between N virtual terminals and the server via the network, where N is an integer greater than or equal to 1, and each virtual terminal is equipped with a camera. For any one of the N virtual terminals, the RGB image captured by the camera is obtained, and the SLAM technology is used to extract the 2D visual features and 3D point cloud features in the image. Based on the extracted 2D visual features and 3D point cloud features, a visual map is generated and transmitted to the server via the network. Among them, the SLAM technology can be implemented based on a simple SLAM system, or based on a visual and inertial VO-SLAM system. The process of generating the visual map is already quite mature and is not the focus of this application, so it will not be elaborated on in detail here.

[0053] The server uses map fusion technology to match and align the coordinate systems of multiple visual maps based on the key frames corresponding to each visual map, and fuses the visual maps in the common viewing area into a new visual map. Furthermore, based on the fused new visual map, the state of virtual objects in the common viewing area of ​​N virtual terminals is synchronized, including the structure, position, posture, action, image and other attributes of the virtual objects, and the fused new visual map is transmitted to the N virtual terminals via the network, so that each virtual terminal updates the original visual map to the new visual map, realizing the sharing of virtual vision. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0054] exist Figure 3 In the embodiment of the present invention, the transmission protocol of the network transmission can be determined according to the data structure of the visual map. For example, a socket can be used for data transmission, or a user datagram protocol (UDP) can be used for data transmission. The embodiment of the present application does not make any restrictive requirements.

[0055] exist Figure 3 In the overall architecture shown, each virtual terminal generates a visual map of its own terminal based on the RGB image captured by its own camera. Taking a virtual terminal as an example, the process of generating a visual map can be seen in Figure 4 , mainly includes the following steps:

[0056] S401: The virtual terminal obtains a set of RGB images of a real scene captured by a camera.

[0057] In the process of generating the visual map, the virtual terminal continuously moves in the real scene, and the camera collects RGB images of the real scene at a set frame rate to obtain an RGB image set, and transmits the RGB images in the RGB image set to the SLAM system in the virtual terminal in real time.

[0058] S402: The virtual terminal selects a plurality of key frames from the RGB image set.

[0059] Since the virtual terminal is in a constant state of motion, different RGB images will be collected. In this way, the SLAM system can filter out multiple RGB images with rich feature information from the RGB image collection as key frames based on the number of corner points and visual features contained in the RGB images.

[0060] S403: For each key frame, the virtual terminal extracts 2D feature points and 3D point clouds in the key frame.

[0061] Each 3D point in the 3D point cloud contains the depth information of the 2D feature point. The embodiment of the present application does not impose any restrictions on the extraction algorithm of the 2D feature point, and the SIFT algorithm, SURF algorithm or ORB algorithm can be used.

[0062] S404: The virtual terminal generates an initial visual map based on the 2D feature points and 3D point clouds extracted from the multiple key frames.

[0063] The process of generating visual maps using SLAM systems is quite mature and is not the focus of this application, so it will not be elaborated in detail here.

[0064] After multiple virtual terminals have reconstructed their own initial visual maps, they are sent to the server, which then performs map fusion on the multiple initial visual maps.

[0065] See also Figure 5 , a method process for sharing virtual vision based on the fusion technology of multiple visual maps is provided for the embodiment of the present application. The process is executed by the server and mainly includes the following steps:

[0066] S501: receiving initial visual maps sent by multiple virtual terminals, and determining a reference visual map from the multiple initial visual maps.

[0067] In an optional embodiment, when executing S501, multiple virtual terminals in the same area send their respective initial visual maps to the server respectively. The server uses the initial visual map received first as the reference visual map and the other initial visual maps as non-reference visual maps, and determines its own map coordinate system based on the reference visual map. That is, the map coordinate system of the server is consistent with the coordinate system of the virtual terminal that constructs the reference visual map.

[0068] In an embodiment of the present application, after receiving multiple initial visual maps, the server generates a historical key frame library based on the key frames contained in the multiple initial visual maps, and uses a bag of words (BOW) algorithm to extract historical visual vocabulary from each historical key frame in the historical key frame library to establish a visual word bag library.

[0069] S502: For each key frame in any non-reference visual map, determine whether the key frame has a common viewing area with the historical key frames in the historical key frame library. If so, execute S503; if not, obtain the next key frame in the current non-reference visual map.

[0070] After the server determines the reference visual map, for each keyframe in any non-reference visual map, it can determine whether the keyframe has a common viewing area with the historical keyframes based on the 2D feature points and visual word bag library extracted from the keyframe. For the specific implementation process, see Figure 6 :

[0071] S5021: Extract 2D feature points and at least one visual word from the key frame.

[0072] In an optional implementation, an ORB algorithm using a BREIF feature descriptor is used to extract 2D feature points in the key frame, and a BOW algorithm is used to obtain at least one visual vocabulary of the key frame.

[0073] like Figure 7 As shown in FIG, the solid dots represent the 2D feature points extracted from the key frame, and the rectangular boxes are the image features represented by the extracted visual vocabulary.

[0074] S5022: Use at least one visual word to query a pre-established bag-of-visual-words database, and determine a corresponding historical key frame according to the at least one historical visual word found.

[0075] When a historical visual word that is the same as a visual word is found in the bag of visual words library, it indicates that the key frame corresponding to the visual word and the historical key frame corresponding to the historical visual word may have a common viewing area, and the more historical visual words are found, the greater the possibility.

[0076] S5023: Extract the 2D feature points in the obtained historical key frame and perform feature matching with the 2D feature points in the key frame.

[0077] In S5023, the same algorithm as that used to extract 2D feature points in the key frame is used to extract 2D feature points in the historical key frame, and the RANSAC (Random Sample Consensus) algorithm is used to perform feature matching on the 2D feature points in the historical key frame and the 2D feature points in the key frame.

[0078] S5024: Determine whether the matching degree is greater than a preset matching threshold. If so, execute S5025; otherwise, execute S5026.

[0079] In S5024 , it is determined whether there is a common view area based on a comparison result between the matching degree and a preset matching threshold.

[0080] S5025: Determine whether the key frame has a common viewing area with the historical key frames obtained from the historical key frame library.

[0081] When the key frame and the historical key frame have the same visual vocabulary and the matching degree of 2D feature points is high, it can be determined that the key frame and the historical key frame obtained from the historical key frame library have a common viewing area.

[0082] S5026: Determine that there is no common viewing area between the key frame and the historical key frames obtained from the historical key frame library.

[0083] When the matching degree between the key frame and the 2D feature points in the historical key frame is low, it can be determined that the key frame and the historical key frame obtained from the historical key frame library do not have a common viewing area.

[0084] like Figure 8 As shown, there are 2D feature points (represented by solid small dots) extracted from the historical key frames in the common viewing area with the key frame, and the image features represented by the historical visual vocabulary (circled in a rectangular box).

[0085] S503: Determine whether the key frame and the historical key frame with a common viewing area belong to different virtual terminals. If they belong to different virtual terminals, execute S504; if they belong to the same virtual terminal, execute S506.

[0086] Since the historical key frame library is generated based on the key frames contained in the multiple received initial visual maps, the key frame and the historical key frame with a common view area may come from the same non-reference visual map, or may come from a non-reference visual map and a reference visual map respectively. In other words, the key frame and the historical key frame with a common view area may belong to the same virtual terminal, or may belong to different virtual terminals.

[0087] S504: performing map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the obtained historical key frame to obtain a target visual map.

[0088] When the key frame and the historical key frame with a common view area belong to different virtual terminals, the non-reference visual map corresponding to the key frame can be fused with the reference visual map corresponding to the historical key frame. For the specific fusion process, see Figure 9 :

[0089] S5041: Perform feature matching between the key frame and the target 3D point cloud extracted from the obtained historical key frames.

[0090] In an embodiment of the present application, by matching the 3D point cloud between the key frame and the historical key frame in the common view area, the pose matrix between two different virtual terminals can be determined, thereby aligning the map coordinate systems between the two virtual terminals for map fusion.

[0091] The specific matching process of 3D point cloud can be found in Figure 10 :

[0092] S5041_1: Determine the key frame and the obtained historical key frame as the center of the non-reference visual map and the reference visual map respectively, and obtain two spheres with preset radius.

[0093] like Figure 11 As shown, the key frame is determined as the center of the non-reference visual map. Figure 11 It is represented by a solid dot in the middle, with the key frame as the center and the preset length N as the radius, to determine the first sphere. Figure 11 The solid line in represents the motion trajectory of the virtual terminal corresponding to the non-reference visual map.

[0094] Similarly, the historical key frame is determined as the center of the reference visual map. Figure 12 It is also represented by a solid circle, with the historical key frame as the center and the preset length N as the radius to determine the second sphere. Figure 12 The solid line in represents the motion trajectory of the virtual terminal corresponding to the reference visual map.

[0095] It should be noted that Figure 11 and Figure 12 A solid dot is used to represent the key frame and the historical key frame because the key frame and the historical key frame are a frame in the non-reference visual map and the reference visual map respectively, but it does not mean that there is only one 3D point in the key frame and the historical key frame.

[0096] S5041_2: Obtain the initial 3D point clouds within two spherical regions.

[0097] In S5041_2, the inner 3D point cloud of the first sphere is used as the initial 3D point cloud of the key frame. Figure 11 At the same time, the 3D point cloud in the second sphere is used as the initial 3D point cloud of the historical key frame. Figure 12 It is also represented by a triangle.

[0098] S5041_3: Perform feature matching on the 2D feature points in the key frame and the 2D feature points in the historical key frames, and filter the two initial 3D point clouds based on the matching results to obtain two target 3D point clouds.

[0099] In the embodiment of the present application, 2D feature points can be described by BREIF feature descriptors, because BREIF features can represent the data structure of 3D points, such as Figure 13 As shown, the initial 3D point cloud can be screened using the matching results of the 2D feature points.

[0100] In the specific implementation, the RANSAC algorithm is used to match the 2D feature points in the key frame and the historical key frame to obtain similar point pairs in the key frame and the historical key frame. Based on the similar point pairs, the two initial 3D point clouds are filtered to remove the noise in the initial 3D point cloud and obtain two target 3D point clouds. The target 3D point cloud in the key frame before and after the screening and the target 3D point cloud in the historical key frame are respectively as follows: Figure 14 and Figure 15 As shown, the target 3D point cloud in the key frame is represented by a solid triangle, and the target 3D point cloud in the historical key frame is represented by a spatial triangle.

[0101] Optionally, the 2D feature points can be described using a BREIF descriptor.

[0102] S5041_4: Perform feature matching on two target 3D point clouds.

[0103] In S5041_4, the Iterative Closest Point (ICP) algorithm is used to perform feature matching on the two target 3D point clouds. Since the two filtered target 3D point clouds have less noise, the matching accuracy is improved, thereby improving the accuracy of the coordinate system alignment.

[0104] S5042: Determine the rotation matrix and translation vector between the key frame and the two virtual terminals to which the historical key frame belongs based on the matched 3D point cloud pair.

[0105] The process of determining the rotation matrix and translation vector based on the matched 3D point cloud pairs is a mature algorithm and will not be described in detail. It can be completed with the help of MATLAB tools or by calling the functions provided by OpenCV.

[0106] S5043: Using the rotation matrix and the translation vector, perform map fusion on the non-reference visual map and the reference visual map to obtain the target visual map.

[0107] The structures of the rotation matrix and translation vector are as follows:

[0108]

[0109] Among them, R represents a rotation matrix with 3 rows and 3 columns, and T represents a translation vector with 3 rows and 1 column.

[0110] Based on the rotation matrix and translation vector, the PnP algorithm is used to transform the 3D points in each keyframe in the non-reference visual map to the map coordinate system corresponding to the reference visual map, thereby aligning the map coordinate systems of different virtual terminals. The conversion formula is as follows:

[0111]

[0112] in, Represents the pose matrix of the transformed 3D point, Represents the pose matrix of the 3D point before transformation.

[0113] After the map coordinate system is aligned, the non-reference visual map and the reference visual map are fused to obtain a target visual map. The map trajectories of the two different virtual terminals after fusion are as follows: Figure 16 shown.

[0114] S505: Synchronize the status information of the virtual objects in the common viewing area according to the target visual map, and send the status information and the target visual map to the corresponding virtual terminals respectively to realize the sharing of virtual vision.

[0115] The state information of the virtual object includes but is not limited to the structure, position, posture, action, image and other attributes of the virtual object. After synchronizing the virtual objects that are viewed by different virtual terminals, the state information of the virtual object and the target visual map are sent to the virtual terminal corresponding to the non-reference visual map and the virtual terminal corresponding to the reference visual map respectively. In this way, the two virtual terminals can realize the sharing of virtual vision. The effect after sharing virtual vision is as follows: Figure 17 shown.

[0116] In an optional embodiment, the server obtains the target visual map and updates the target visual map to the reference visual map. When a new virtual terminal is connected to the server during the interaction process, the initial visual map of the newly connected virtual terminal is used as the non-reference visual map. Figure 5 The non-reference visual map and the reference visual map are fused in the manner shown, so that the map coordinate system of the newly accessed terminal can be guaranteed to be consistent with the coordinate system of the server, such as Figure 18As shown, the newly connected virtual terminal can share the virtual field of view with the already connected virtual terminal.

[0117] Each non-reference visual map is fused with the reference visual map, so that the map coordinate systems of multiple virtual terminals are unified under the map coordinate system of one virtual terminal, thereby realizing the virtual field of view sharing of multiple virtual terminals, that is, seeing virtual objects with the same posture at the same position in the real world, such as Figure 19 shown.

[0118] Optionally, in some embodiments, after obtaining the target visual map, the server uses a global bundle adjustment method to optimize the target visual map to further improve the common view accuracy of the virtual field of view.

[0119] S506: Perform loop closure check on the current non-reference visual map based on the key frame and the obtained historical key frames.

[0120] When the key frame and the historical key frame obtained from the historical key frame library both come from the current non-reference visual map, that is, the key frame and the historical key frame belong to the same virtual terminal, SLAM technology is used to perform loop verification on the current non-reference visual map to update the current non-reference visual map.

[0121] Optionally, in order to improve the accuracy of the non-reference visual map, a global bundle adjustment method may be used to optimize the current non-reference visual map after loopback verification, and the optimized current non-reference visual map may be synchronized to the corresponding virtual terminal.

[0122] In an embodiment of the present application, a method for sharing a virtual field of view based on visual map fusion is provided. The server selects a reference visual map from the initial visual maps sent by multiple virtual terminals. For each key frame in any non-reference visual map, 3D point cloud matching is performed with the historical key frames in the reference visual map where the common view area exists, and the rotation matrix and translation vector between the two virtual terminals are determined. The rotation matrix and translation vector are used to align the coordinate systems of the two terminals corresponding to the non-reference visual map and the reference visual map. Based on the selected reference visual map, the server aligns the coordinate systems of the multiple virtual terminals to the map coordinate system of the same virtual terminal, and ensures that the coordinate system of each virtual terminal is consistent with the map coordinate system of the server. Further, the non-reference visual map after coordinate system alignment is fused with the reference visual map. The fused target visual map synchronizes the status information of the virtual objects in the virtual field of view shared by each virtual terminal. In this way, after the fused target visual map and status information are sent to each virtual terminal, each virtual terminal can accurately and efficiently share the virtual field of view.

[0123] Based on the same technical concept, an embodiment of the present application provides a server that can implement the method steps of sharing virtual vision based on visual map fusion in the above embodiment and achieve the same technical effect.

[0124] See also Figure 20 , the server includes a processor 2001, a memory 2002 and a communication interface 2003, wherein the communication interface 2003, the memory 2002 and the processor 2001 are connected via a bus 2004;

[0125] The memory 2002 stores a computer program, and the processor 2001 performs the following operations according to the computer program:

[0126] Receiving initial visual maps sent by multiple virtual terminals through the communication interface 2003, and determining a reference visual map from the multiple initial visual maps;

[0127] For each keyframe in any non-reference visual map, determining whether the keyframe has a common view area with a historical keyframe in a historical keyframe library, where the historical keyframe library is generated based on the keyframes included in the multiple initial visual maps;

[0128] When there is a common viewing area, determining whether the key frame and the historical key frame belong to different virtual terminals;

[0129] When belonging to different virtual terminals, performing map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame to obtain a target visual map;

[0130] The state information of the virtual objects in the common viewing area is synchronized according to the target visual map, and the state information and the target visual map are sent to the corresponding virtual terminals through the communication interface 2003 to realize the sharing of virtual vision.

[0131] Optionally, the processor 2001 determines whether there is a common viewing area between the key frame and the historical key frame by:

[0132] extracting 2D feature points and at least one visual vocabulary from the keyframe;

[0133] Using the at least one visual vocabulary to query a pre-established bag-of-visual-words database, and determining a corresponding historical keyframe based on the at least one historical visual vocabulary found in the query;

[0134] Extract the 2D feature points in the historical key frame and perform feature matching with the 2D feature points in the key frame; if the matching degree is greater than a preset matching threshold, determine that the key frame and the historical key frame have a common viewing area.

[0135] Optionally, the processor 2001 performs map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame to obtain a target visual map. The specific operations are:

[0136] Performing feature matching on the key frame and the target 3D point cloud extracted from the historical key frame;

[0137] Determine, based on the matched 3D point cloud pair, a rotation matrix and a translation vector between the key frame and two virtual terminals to which the historical key frame belongs;

[0138] The non-reference visual map and the reference visual map are fused using the rotation matrix and the translation vector to obtain a target visual map.

[0139] Optionally, the processor 2001 performs feature matching on the key frame and the target 3D point cloud extracted from the historical key frame, specifically by:

[0140] Determining the key frame and the historical key frame as the centers of the non-reference visual map and the reference visual map, respectively, to obtain two spheres with preset radiuses;

[0141] Get the initial 3D point clouds within two spherical regions;

[0142] Performing feature matching on the 2D feature points in the key frame and the 2D feature points in the historical key frame, and screening two initial 3D point clouds according to the matching results to obtain two target 3D point clouds;

[0143] Perform feature matching on two target 3D point clouds.

[0144] Optionally, after obtaining the target visual map and before sending the target visual map to the corresponding virtual terminals, the processor 2001 further executes:

[0145] The target visual map is optimized using a global bundle adjustment method.

[0146] Optionally, when the key frame and the historical key frame belong to the same virtual terminal, the processor 2001 further executes:

[0147] performing a loop closure check on the non-reference visual map according to the key frame and the historical key frame;

[0148] The global bundle adjustment method is used to optimize the non-reference visual map after loop closure verification.

[0149] Optionally, each initial visual map is generated based on 2D feature points and 3D point clouds extracted from multiple key frames collected by the corresponding virtual terminal.

[0150] Embodiments of the present application Figure 20 The processor involved may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The memory may be integrated into the processor or may be provided separately from the processor.

[0151] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0152] An embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0153] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0157] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for sharing virtual vision based on visual map fusion, characterized in that: include: receiving initial visual maps sent by multiple virtual terminals, and determining a reference visual map from the multiple initial visual maps; For each keyframe in any non-reference visual map, determining whether the keyframe has a common view area with a historical keyframe in a historical keyframe library, where the historical keyframe library is generated based on the keyframes included in the multiple initial visual maps; When there is a common viewing area, determining whether the key frame and the historical key frame belong to different virtual terminals; When belonging to different virtual terminals, feature matching is performed on the target 3D point cloud extracted from the key frame and the historical key frame, and based on the matched 3D point cloud pairs, map fusion is performed on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame to obtain the target visual map; Synchronizing state information of virtual objects in the common viewing area according to the target visual map, and sending the state information and the target visual map to corresponding virtual terminals respectively to achieve sharing of virtual vision; The step of performing feature matching on the key frame and the target 3D point cloud extracted from the historical key frame includes: Determining the key frame and the historical key frame as the centers of the non-reference visual map and the reference visual map, respectively, to obtain two spheres with preset radiuses; Get the initial 3D point clouds within two spherical regions; Performing feature matching on the 2D feature points in the key frame and the 2D feature points in the historical key frame, and screening two initial 3D point clouds according to the matching results to obtain two target 3D point clouds; Perform feature matching on two target 3D point clouds.

2. The method according to claim 1, wherein Determine whether the key frame and the historical key frame have a common viewing area by: extracting 2D feature points and at least one visual vocabulary from the keyframe; Using the at least one visual vocabulary to query a pre-established bag-of-visual-words database, and determining a corresponding historical keyframe based on the at least one historical visual vocabulary found in the query; Extracting 2D feature points from the historical key frames and performing feature matching with the 2D feature points in the key frames; If the matching degree is greater than a preset matching threshold, it is determined that the key frame and the historical key frame have a common viewing area.

3. The method according to claim 1, wherein The step of performing map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame according to the matched 3D point cloud pair to obtain a target visual map includes: Determine, based on the matched 3D point cloud pair, a rotation matrix and a translation vector between the key frame and two virtual terminals to which the historical key frame belongs; The non-reference visual map and the reference visual map are fused using the rotation matrix and the translation vector to obtain a target visual map.

4. The method according to claim 1, wherein After obtaining the target visual map and before sending the target visual map to the corresponding virtual terminals, the method further includes: The target visual map is optimized using a global bundle adjustment method.

5. The method according to claim 1, wherein When the key frame and the historical key frame belong to the same virtual terminal, the method further includes: performing a loop closure check on the non-reference visual map according to the key frame and the historical key frame; The global bundle adjustment method is used to optimize the non-reference visual map after loop closure verification.

6. The method according to any one of claims 1 to 4, wherein Each initial visual map is generated based on the 2D feature points and 3D point clouds extracted from multiple key frames collected by the corresponding virtual terminal.

7. A server, characterized in that: comprising a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: receiving, via the communication interface, initial visual maps sent by a plurality of virtual terminals, and determining a reference visual map from the plurality of initial visual maps; For each keyframe in any non-reference visual map, determining whether the keyframe has a common view area with a historical keyframe in a historical keyframe library, where the historical keyframe library is generated based on the keyframes included in the multiple initial visual maps; When there is a common viewing area, determining whether the key frame and the historical key frame belong to different virtual terminals; When belonging to different virtual terminals, feature matching is performed on the target 3D point cloud extracted from the key frame and the historical key frame, and based on the matched 3D point cloud pairs, map fusion is performed on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame to obtain the target visual map; Synchronizing state information of virtual objects in the common viewing area according to the target visual map, and sending the state information and the target visual map to corresponding virtual terminals through the communication interface to achieve sharing of virtual vision; The feature matching is performed on the target 3D point cloud extracted from the key frame and the historical key frame, and the specific operation is: Determining the key frame and the historical key frame as the centers of the non-reference visual map and the reference visual map, respectively, to obtain two spheres with preset radiuses; Get the initial 3D point clouds within two spherical regions; Performing feature matching on the 2D feature points in the key frame and the 2D feature points in the historical key frame, and screening two initial 3D point clouds according to the matching results to obtain two target 3D point clouds; Perform feature matching on two target 3D point clouds.

8. The server according to claim 7, wherein: The processor determines whether there is a common viewing area between the key frame and the historical key frame by: extracting 2D feature points and at least one visual vocabulary from the keyframe; Using the at least one visual vocabulary to query a pre-established bag-of-visual-words database, and determining a corresponding historical keyframe based on the at least one historical visual vocabulary found in the query; Extracting 2D feature points from the historical key frames and performing feature matching with the 2D feature points in the key frames; If the matching degree is greater than a preset matching threshold, it is determined that the key frame and the historical key frame have a common viewing area.

9. The server according to claim 7, wherein: The processor performs map fusion on the non-reference visual map corresponding to the key frame and the reference visual map corresponding to the historical key frame according to the matched 3D point cloud pair to obtain a target visual map. The specific operations are as follows: Determine, based on the matched 3D point cloud pair, a rotation matrix and a translation vector between the key frame and two virtual terminals to which the historical key frame belongs; The non-reference visual map and the reference visual map are fused using the rotation matrix and the translation vector to obtain a target visual map.

Images