A general method and system for multi-person AR with cross-system collaboration

Abstract

The present application relates to a general method and system for multi-person AR with cross-system collaboration, wherein the method comprises: obtaining a first image frame and a first 6DoF pose through a device end; configuring a room number and a map number on the cloud end, constructing a feature vector according to the first image frame and the first 6DoF pose, optimizing the first 6DoF pose according to the feature vector, completing AR mapping and storing; initializing and loading an AR map on the device end or the cloud end according to the room number and the map number; after completing AR positioning matching according to the second image frame and the second 6DoF pose, obtaining a conversion matrix between the system coordinate system and the map coordinate system, and realizing multi-person AR interaction in the AR map. Through the present application, the problems of lack of portability and insufficient computing resources on the device end in the multi-person AR solution are solved, the 6DoF tracking modules under most different framework systems are adapted, the client-cloud collaborative work is realized, and the computing pressure caused by AR mapping and positioning on the client is reduced.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and in particular to a general method and system for multi-person AR with cross-system collaboration. Background Art

[0002] Multiplayer AR (Augment Reality) solutions are an important development direction of the current AR industry and are widely used in games, military, industry and other scenarios. Multiplayer AR, as the name suggests, is an AR activity conducted by multiple people. That is, in an AR scene, multiple AR devices can know their position and orientation through the multiplayer AR system. Compared with single-player AR, multiplayer AR enables multiple different people to interact in the same AR system, or collaborate with each other, or fight against each other. This greatly raises the upper limit of the AR solution's capabilities and enriches the application scenarios of the AR solution, which is of extraordinary significance to the development and application of AR.

[0003] Traditional multiplayer AR solutions are mostly integrated into a certain established AR system, such as Apple's ARKit, Google's ARCore, Huawei's AREngine, etc. They all have their own set of fixed interfaces and application modes. Multiplayer AR solutions can only be designed according to their interfaces and modes, and can only run in their own systems and devices supported by them. Different multiplayer AR solutions cannot communicate with each other, and are seriously lacking in portability and adaptability. In addition, the multiplayer AR solutions in these AR systems arrange all computing power on devices such as mobile phones, tablets, and VR / AR glasses, which puts huge computing pressure on the device side and easily causes problems such as device overheating and frequency reduction.

[0004] Currently, no effective solution has been proposed to address the problems of lack of portability of multi-person AR solutions and insufficient computing resources on the device side in related technologies. Summary of the invention

[0005] The embodiments of the present application provide a general method and system for multi-person AR with cross-system collaboration, so as to at least solve the problems of lack of portability and insufficient computing resources on the device side of the multi-person AR solution in the related art.

[0006] In a first aspect, an embodiment of the present application provides a general method for multi-person AR collaboration across systems, the method comprising:

[0007] Acquire the first image frame in the continuous time sequence and the corresponding first 6DoF pose in the system coordinate system through the device end;

[0008] Configure the room number and the map number through the cloud, and extract the ORB feature points of the first image frame through the ORB algorithm according to the first image frame and the first 6DoF pose;

[0009] Constructing a feature vector using a preset bag-of-words model library based on the ORB feature points through the cloud;

[0010] Optimizing the first 6DoF posture according to the feature vector through the cloud, completing AR mapping and storing it;

[0011] Initialize and load the AR map according to the room number and the map number;

[0012] After completing AR positioning matching according to the second image frame and the second 6DoF posture, a conversion matrix between the system coordinate system and the map coordinate system is obtained, and multi-person AR interaction is achieved in the AR map.

[0013] In some embodiments, after completing AR positioning matching according to the second image frame and the second 6DoF pose, obtaining a transformation matrix between the system coordinate system and the map coordinate system includes:

[0014] Extracting ORB feature points of the second image frame by using an ORB algorithm according to the second image frame and the second 6DoF pose;

[0015] Constructing a bag-of-words vector and a feature vector according to the ORB feature points through the preset bag-of-words model library;

[0016] According to the bag-of-words vector and the feature vector, positioning matching is performed in the AR map, and if the positioning matching is successful, the camera pose in the map coordinate system is calculated by an efficient perspective n-point image pose estimation method;

[0017] A transformation matrix between the system coordinate system and the map coordinate system is calculated according to the second 6DoF pose and the camera pose.

[0018] In some embodiments, optimizing the first 6DoF pose according to the feature vector through the cloud to complete AR mapping and store includes:

[0019] Performing 2D-2D feature matching on the first image frame using the feature vector in the cloud, and then performing 2D-3D feature matching on the first image frame and the map point;

[0020] After the 2D-2D feature matching and the 2D-3D feature matching are completed, optimizing the first 6DoF pose using a g2o library in the cloud;

[0021] AR mapping is completed through the cloud, and the serialized AR map is stored as a binary map file.

[0022] In some embodiments, performing positioning matching in the AR map according to the bag-of-words vector and the feature vector includes:

[0023] Performing image retrieval in the AR map according to the bag-of-words vector of the second image frame,

[0024] If the corresponding map image is not retrieved, the positioning fails; if the corresponding map image is retrieved, the positioning is successful;

[0025] After successful positioning, feature matching between the second image frame and the map image is accelerated according to the feature vector. If the number of matches reaches a preset threshold, the matching is successful.

[0026] In some embodiments, calculating a transformation matrix between the system coordinate system and the map coordinate system according to the 6DoF pose and the camera pose includes:

[0027] According to the second 6DoF pose and the camera pose By formula Calculate the transformation matrix between the system coordinate system and the map coordinate system Among them, the inverse() function represents the inversion of the matrix.

[0028] In some embodiments, after completing AR mapping and storing, the method further includes:

[0029] Detect AR maps corresponding to different map numbers under the same room number. If the overlapping observation parts between the AR maps are greater than a preset number, merge the AR maps to obtain a fused new AR map, wherein the map number of the new AR map is the smallest map number among the AR maps.

[0030] In some embodiments, initializing loading of the AR map according to the room number and the map number includes:

[0031] Load the location according to the room number and map number Figure 2 Binary file;

[0032] To the said place Figure 2 Deserialize the binary file to obtain the data structure contained in the AR map.

[0033] In some embodiments, before initializing loading of the AR map according to the room number and the map number, the method further includes:

[0034] A second image frame in a continuous time sequence is acquired through the device end, and a second 6DoF posture in the system coordinate is obtained through a preset 6DoF tracking module according to the second image frame.

[0035] In some embodiments, obtaining, by the device end, a first image frame in a continuous time sequence and a corresponding first 6DoF pose in a system coordinate system includes:

[0036] A first image frame in a continuous time sequence is acquired through the device end, and a first 6DoF pose in the system coordinate is obtained through a preset 6DoF tracking module according to the first image frame, wherein the preset 6DoF tracking module includes an ARKit system, an ARCore system, and an AREngine system.

[0037] In a second aspect, an embodiment of the present application provides a multi-person AR general system for cross-system collaboration, the system comprising a device side and a cloud side;

[0038] The device end obtains a first image frame in a continuous time sequence and a corresponding first 6DoF pose in a system coordinate system;

[0039] The cloud configures the room number and the map number, and extracts the ORB feature points of the first image frame by using the ORB algorithm according to the first image frame and the first 6DoF pose;

[0040] The cloud constructs a feature vector using a preset bag-of-words model library according to the ORB feature points;

[0041] The cloud optimizes the first 6DoF pose according to the feature vector, completes AR mapping and stores it;

[0042] The device or the cloud initializes and loads the AR map according to the room number and the map number;

[0043] After the device or the cloud completes AR positioning matching according to the second image frame and the second 6DoF posture, a conversion matrix between the system coordinate system and the map coordinate system is obtained, and multi-person AR interaction is realized in the AR map.

[0044] Compared with the related art, the embodiment of the present application provides a general method and system for multi-person AR with cross-system collaboration, which uses a preset 6DoF tracking module on the device side to obtain a first image frame and a first 6DoF pose; configures a room number and a map number on the cloud side, extracts the ORB feature points of the first image frame according to the first image frame and the first 6DoF pose through the ORB algorithm, and then uses a preset bag-of-words model library to construct a feature vector, optimizes the first 6DoF pose according to the feature vector, completes AR mapping and stores it; the device side or the cloud side initializes and loads the AR map according to the room number and the map number; after completing AR positioning matching according to the second image frame and the second 6DoF pose, a conversion matrix between the system coordinate system and the map coordinate system is obtained, and multi-person AR interaction is realized in the AR map, which solves the problems of lack of portability and insufficient computing resources on the device side in multi-person AR solutions, adapts to the 6DoF tracking modules under most different framework systems, realizes the client-cloud collaborative working mode, and reduces the computing pressure caused by AR mapping and positioning on the client. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 is a flowchart of the steps of a general method for multi-person AR across system collaboration according to an embodiment of the present application;

[0047] Figure 2 It is a flowchart of the general method for implementing dual-view multi-person AR;

[0048] Figure 3 It is a flow chart of the steps for calculating the transformation matrix between the system coordinate system and the map coordinate system;

[0049] Figure 4 is a structural block diagram of a multi-person AR general system for cross-system collaboration according to an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application.

[0051] Description of the accompanying drawings: 41. Device side; 42. Cloud side. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] The present application embodiment provides a general method for multi-person AR collaboration across systems. Figure 1 is a flowchart of the steps of the general method for multi-person AR across system collaboration according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0057] Step S102, obtaining, through the device end, a first image frame in a continuous time sequence and a first 6DoF pose corresponding to the system coordinate system;

[0058] It should be noted that the first image frame is a series of images acquired in a continuous time sequence, and step S102 completes the 6DoF tracking part of the multi-person AR general method in this embodiment, 6DoF (6 Degree of Freedom, 6 degrees of freedom, 3 rotational degrees of freedom + 3 displacement degrees of freedom).

[0059] Step S104, configuring the room number and the map number through the cloud, and extracting the ORB feature points of the first image frame through the ORB algorithm according to the first image frame and the first 6DoF pose;

[0060] Specifically, configure the room number through the cloud, and then configure the map number (a device can only build one map at a time, and multiple devices can build multiple maps in one room or in different rooms at the same time);

[0061] According to the first image frame and the first 6DoF pose, the ORB feature points of the first image frame are extracted by using the ORB algorithm (Oriented FAST and RotatedBRIEF).

[0062] Step S106, constructing a feature vector using a preset bag-of-words model library according to the ORB feature points through the cloud;

[0063] Specifically, the feature vector (FeatureVector) is constructed through the cloud based on the ORB feature points using the bag-of-words model library DBoW3;

[0064] Step S108, optimizing the first 6DoF pose according to the feature vector through the cloud, completing AR mapping and storing it;

[0065] It should be noted that steps S104 to S108 complete the AR mapping part of the multi-person AR general method in this embodiment. The execution body is the cloud, that is, the mapping module is placed in the cloud. Putting the mapping module (the module that occupies the most computing resources and has a slower mapping speed on the device side) into the cloud for computing solves the problem of limited computing power on the device side, effectively reduces the occurrence of overheating, frequency reduction and freezes on the device side, and realizes a collaborative working mode between the client and the cloud.

[0066] Step S110, initializing and loading the AR map according to the room number and the map number;

[0067] Step S112, after completing AR positioning matching according to the second image frame and the second 6DoF posture, a conversion matrix between the system coordinate system and the map coordinate system is obtained, and multi-person AR interaction is realized in the AR map.

[0068] It should be noted that the second image frame is a series of images acquired in a continuous time sequence. Steps S110 and S112 complete the AR positioning part of the multi-person AR general method in this embodiment. The execution subject can be the device side or the cloud. In the case where the execution subject is the device side, that is, the positioning module is placed on the device side (mobile terminals with AR functions such as mobile phones, tablets and AR glasses), it will occupy the computing resources of the device side, and the AR map file needs to be loaded from the cloud to the device side. The loading speed is slightly slower, but at the same time there is no network delay and no frame loss will occur; in the case where the execution subject is the cloud, that is, the positioning module is placed on the cloud server, it does not occupy the computing resources of the device side, and the AR map file is directly loaded from the cloud server disk. The loading speed is fast, and the device side needs to package the information output by the 6DoF module such as images and postures according to the network transmission protocol and transmit it to the cloud, and then the cloud performs the calculation.

[0069] It can be seen from the above steps S102 to S112 that the multi-person AR general method in this embodiment mainly includes an AR mapping part and an AR positioning part. Figure 2 This is a flowchart of the general method for implementing dual-view multi-person AR. Figure 2 As shown in the figure, the implementation process of the mapping and positioning stages of this method is presented from the perspective of interaction and algorithm.

[0070] Through steps S102 to S112 in the embodiment of the present application, the problems of lack of portability and insufficient computing resources on the device side in the multi-person AR solution are solved, the 6DoF tracking module under most different framework systems is adapted, the client-cloud collaborative working mode is realized, and the computing pressure caused by AR mapping and positioning on the client is reduced.

[0071] In some of these embodiments, Figure 3 It is a flowchart of the steps to calculate the conversion matrix between the system coordinate system and the map coordinate system, such as Figure 3 As shown, in step S112, after completing AR positioning matching according to the second image frame and the second 6DoF posture, the conversion matrix between the system coordinate system and the map coordinate system is obtained, including:

[0072] Step S302, extracting ORB feature points of the image frame by using an ORB algorithm according to the second image frame and the second 6DoF pose;

[0073] Specifically, according to the second image frame and the second 6DoF pose, ORB feature points of the second image frame are extracted by using an ORB algorithm (Oriented FAST and Rotated BRIEF).

[0074] Step S304, constructing a bag-of-words vector and a feature vector according to the ORB feature points through a preset bag-of-words model library;

[0075] Specifically, the bag-of-words model library DBoW3 is used in the cloud to construct bag-of-words vectors (BowVector) and feature vectors (FeatureVector) based on ORB feature points. The bag-of-words vector is used for image retrieval and scoring, and similar images will have higher scores. The feature vector is used to accelerate feature matching.

[0076] Step S306, performing positioning matching in the AR map according to the bag-of-words vector and the feature vector. If the positioning matching is successful, the camera pose in the map coordinate system is calculated by an efficient perspective n-point image pose estimation method.

[0077] Specifically, positioning matching is performed in the AR map based on the bag-of-words vector and the feature vector. If the positioning matching is successful, the position and posture of the current image (the second image frame) in the map are calculated using the Efficient Perspective-n-Point Camera Pose Estimation (EPnP) method to obtain the camera pose in the map coordinate system.

[0078] Step S308: Calculate the transformation matrix between the system coordinate system and the map coordinate system according to the second 6DoF pose and the camera pose.

[0079] In some embodiments, step S108, optimizing the first 6DoF pose according to the feature vector through the cloud, completing AR mapping and storing includes:

[0080] Performing 2D-2D feature matching on the first image frame using feature vectors in the cloud, and then performing 2D-3D feature matching on the first image frame and map points;

[0081] After 2D-2D feature matching and 2D-3D feature matching are completed, the first 6DoF pose is optimized using the g2o library in the cloud;

[0082] AR mapping is completed through the cloud, and the serialized AR map is stored as a binary map file.

[0083] Specifically, the second image frame is a series of images acquired in a continuous time sequence, and the cloud uses feature vectors to perform 2D-2D feature matching on the current image and the previous image frame in the first image frame one by one. After the 2D-2D feature matching is successful, the first image frame and the map point are then matched with 2D-3D features, where the map point represents a point in the map coordinate system.

[0084] After both 2D-2D feature matching and 2D-3D feature matching are successful, the first 6DoF pose is optimized using the g2o (General Graph Optimization) library in the cloud;

[0085] Continuously build maps through the cloud and wait for user instructions to stop building maps, that is, AR mapping is completed. The serialized AR map is saved as a binary map file in the cloud server.

[0086] In some embodiments, step S306, performing positioning matching in the AR map according to the bag-of-words vector and the feature vector includes:

[0087] Perform image retrieval in the AR map based on the bag-of-words vector of the second image frame;

[0088] If the corresponding map image is not retrieved, the positioning fails; if the corresponding map image is retrieved, the positioning is successful;

[0089] After successful positioning, feature matching between the second image frame and the map image is accelerated according to the feature vector. If the number of matches reaches a preset threshold, the match is successful.

[0090] In some embodiments, step S308, calculating a transformation matrix between the system coordinate system and the map coordinate system according to the second 6DoF pose and the camera pose includes:

[0091] According to the second 6DoF pose and camera pose By formula Calculate the transformation matrix between the system coordinate system and the map coordinate system Among them, the inverse() function represents the inversion of the matrix.

[0092] In some embodiments, in step S108, after completing AR mapping and storing, the following steps are further included:

[0093] Detect AR maps corresponding to different map numbers under the same room number. If the overlapping observation part between the AR maps is greater than a preset number, the AR maps are fused to obtain a fused new AR map, where the map number of the new AR map is the smallest map number in the AR maps.

[0094] In some embodiments, step S110, initializing loading of the AR map according to the room number and the map number includes:

[0095] Load the location according to the room number and map number Figure 2 Binary file;

[0096] To the ground Figure 2 Deserialize the binary file to obtain the data structure contained in the AR map.

[0097] In some embodiments, in step S110, before initializing loading the AR map according to the room number and the map number, the following steps are further included:

[0098] A second image frame in a continuous time sequence is acquired through the device end, and a second 6DoF posture in the system coordinate is obtained according to the second image frame through a preset 6DoF tracking system.

[0099] In some embodiments, step S102, obtaining, by the device end, a first image frame in a continuous time sequence and a corresponding first 6DoF pose in a system coordinate system includes:

[0100] The first image frame in a continuous time sequence is acquired through the device end, and the first 6DoF pose in the system coordinate is obtained through a preset 6DoF tracking module according to the first image frame, wherein the preset 6DoF tracking module includes an ARKit system, an ARCore system, and an AREngine system.

[0101] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0102] The present application embodiment provides a multi-person AR universal system for cross-system collaboration. Figure 4 is a structural block diagram of a multi-person AR general system for cross-system collaboration according to an embodiment of the present application, such as Figure 4 As shown, the system includes a device end 41 and a cloud end 42;

[0103] The device 41 obtains the first image frame in the continuous time sequence and the first 6DoF pose corresponding to the system coordinate system

[0104] The cloud 42 configures the room number and the map number, and extracts the ORB feature points of the first image frame through the ORB algorithm according to the first image frame and the first 6DoF pose;

[0105] Cloud 42 constructs feature vectors based on ORB feature points using a preset bag-of-words model library;

[0106] The cloud 42 optimizes the first 6DoF pose according to the feature vector, completes AR mapping and stores it;

[0107] The device 41 or the cloud 42 initializes and loads the AR map according to the room number and the map number;

[0108] After the device end 41 or the cloud end 42 completes the AR positioning matching according to the second image frame and the second 6DoF posture, a conversion matrix between the system coordinate system and the map coordinate system is obtained, and multi-person AR interaction is realized in the AR map.

[0109] It should be noted that the cloud in the embodiment of the present application can use MYSQL to search for files in the database, and the information transmission between the device and the cloud can be encoded in base64 and transmitted to each other through the HTTP protocol.

[0110] Through the device side 41 and the cloud side 42 in the embodiment of the present application, the problems of lack of portability and insufficient computing resources on the device side in the multi-person AR solution are solved, the 6DoF tracking module under most different framework systems is adapted, the client-cloud collaborative working mode is realized, and the computing pressure caused by AR mapping and positioning on the client is reduced.

[0111] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0112] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0113] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0114] In addition, in combination with the cross-system collaborative multi-person AR general method in the above embodiments, the present application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by the processor, any of the cross-system collaborative multi-person AR general methods in the above embodiments is implemented.

[0115] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a general method of multi-person AR for cross-system collaboration is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0116] In one embodiment, Figure 5 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 5 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 5As shown. The electronic device includes a processor, a network interface, an internal memory and a non-volatile memory connected by an internal bus, wherein the non-volatile memory stores an operating system, a computer program and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with an external terminal through a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, and when the computer program is executed by the processor, a general method of multi-person AR with cross-system collaboration is realized, and the database is used to store data.

[0117] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0119] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A general method for multi-person AR with cross-system collaboration, characterized in that: The method comprises: Acquire, through the device end, a first image frame in a continuous time sequence and a corresponding first 6DoF pose in a system coordinate system, wherein the first image frame is a series of images acquired in a continuous time sequence; Configure the room number and the map number through the cloud, and extract the ORB feature points of the first image frame through the ORB algorithm according to the first image frame and the first 6DoF pose; Constructing a feature vector using a preset bag-of-words model library based on the ORB feature points through the cloud; Using the feature vector in the cloud, one by one, perform 2D-2D feature matching on the current image and the previous frame in the first image frame, and then perform 2D-3D feature matching on the first image frame and the map point; after the 2D-2D feature matching and the 2D-3D feature matching are completed, the first 6DoF posture is optimized using the g2o library in the cloud; AR mapping is completed in the cloud, and the serialized AR map is stored as a binary map file; Acquire a second image frame in a continuous time sequence through the device end, and obtain a second 6DoF pose in the system coordinates through a preset 6DoF tracking module according to the second image frame, wherein the second image frame is a series of images acquired in a continuous time sequence; Initialize and load the AR map according to the room number and the map number; According to the second image frame and the second 6DoF pose, the ORB feature points of the second image frame are extracted by the ORB algorithm; according to the ORB feature points, a bag-of-words vector and a feature vector are constructed by the preset bag-of-words model library; according to the bag-of-words vector of the second image frame, an image is retrieved in the AR map, if the corresponding map image is not retrieved, the positioning fails, if the corresponding map image is retrieved, the positioning is successful; after successful positioning, the feature matching between the second image frame and the map image is accelerated according to the feature vector of the second image frame, if the number of matches reaches a preset threshold, the match is successful, and after successful matching, the camera pose in the map coordinate system is calculated by an efficient perspective n-point image pose estimation method; according to the second 6DoF pose and the camera pose, the transformation matrix between the system coordinate system and the map coordinate system is calculated to realize multi-person AR interaction in the AR map.

2. The method according to claim 1, characterized in that: Calculating a transformation matrix between the system coordinate system and the map coordinate system according to the second 6DoF pose and the camera pose includes: According to the second 6DoF pose and the camera pose By formula Calculate the transformation matrix between the system coordinate system and the map coordinate system in, Indicates the second 6DoF pose Perform matrix inversion.

3. The method according to claim 1, characterized in that After completing the AR mapping and storing it, the method further includes: Detect AR maps corresponding to different map numbers under the same room number. If the overlapping observation parts between the AR maps are greater than a preset number, merge the AR maps to obtain a fused new AR map, wherein the map number of the new AR map is the smallest map number among the AR maps.

4. The method according to claim 1, characterized in that: Initializing and loading the AR map according to the room number and the map number includes: Load the map binary file according to the room number and map number; The map binary file is deserialized to obtain the data structure contained in the AR map.

5. The method according to claim 1, characterized in that Acquiring the first image frame in the continuous time sequence and the first 6DoF pose corresponding to the system coordinate system through the device side includes: A first image frame in a continuous time sequence is acquired through the device end, and a first 6DoF pose in the system coordinate is obtained through a preset 6DoF tracking module according to the first image frame, wherein the preset 6DoF tracking module includes an ARKit system, an ARCore system, and an AREngine system.

6. A multi-person AR general system with cross-system collaboration, characterized in that: The system includes a device side and a cloud side; The device end obtains a first image frame in a continuous time sequence and a corresponding first 6DoF pose in a system coordinate system, wherein the first image frame is a series of images obtained in a continuous time sequence; The cloud configures the room number and the map number, and extracts the ORB feature points of the first image frame by using the ORB algorithm according to the first image frame and the first 6DoF pose; The cloud constructs a feature vector using a preset bag-of-words model library according to the ORB feature points; The cloud uses the feature vector to perform 2D-2D feature matching on the current image and the previous image frame in the first image frame one by one, and then performs 2D-3D feature matching on the first image frame and the map point; after the 2D-2D feature matching and the 2D-3D feature matching are completed, the first 6DoF posture is optimized by using the g2o library on the cloud; AR mapping is completed by the cloud, and the serialized AR map is stored as a binary map file; The device end acquires a second image frame in a continuous time sequence, and obtains a second 6DoF posture in the system coordinates through a preset 6DoF tracking module according to the second image frame, wherein the second image frame is a series of images acquired in a continuous time sequence; The device or the cloud initializes and loads the AR map according to the room number and the map number; The device or the cloud extracts ORB feature points of the second image frame through the ORB algorithm based on the second image frame and the second 6DoF pose; constructs a bag-of-words vector and a feature vector through the preset bag-of-words model library based on the ORB feature points; performs image retrieval in the AR map based on the bag-of-words vector of the second image frame, and if the corresponding map image is not retrieved, the positioning fails; if the corresponding map image is retrieved, the positioning is successful; after successful positioning, accelerate the feature matching between the second image frame and the map image based on the feature vector of the second image frame, and if the number of matches reaches a preset threshold, the match is successful, and after the match is successful, the camera pose in the map coordinate system is calculated by an efficient perspective n-point image pose estimation method; based on the second 6DoF pose and the camera pose, calculate the transformation matrix between the system coordinate system and the map coordinate system to realize multi-person AR interaction in the AR map.

Images