Method for constructing global map and electronic equipment

Abstract

The invention provides a method for constructing a global map and electronic equipment, which are used for constructing the global map in a large space, are suitable for various types of AR / VR (Auto Register / Virtual Reality) equipment, have universality and save cost. Comprising the following steps: in response to a received global map construction instruction, determining camera poses respectively corresponding to multiple frames of scene images obtained in advance; carrying out scene image ORB feature extraction on the multiple frames of scene images to obtain respective ORB feature points in the multiple frames of scene images; for each frame of scene image in the multiple frames of scene images, reconstructing the 3D point cloud data of the scene image by using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image; and constructing a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images and the 3D point cloud data corresponding to the multiple frames of scene images.

Description

Background Art

[0002] AR (Augmented Reality) / VR (Virtual Reality) large space solutions continue to develop with the industry. From the development trend, it can be seen that large space positioning technology is gradually commercialized as the cost of AR / VR equipment decreases. Although AR / VR large space solutions are currently available to consumers, the entire solution is still expensive, often costing tens or even hundreds of millions, so the continuous cost reduction of the entire positioning solution is an inevitable trend.

[0003] AR / VR large space positioning is the core technology of the whole solution. In order to ensure the accuracy of virtual-real integration, positioning must maintain high precision. Most of the industry adopts inside-out VR solutions. The inside-out solution relies on the SLAM (Simultaneous Localization and Mapping) algorithm in AR / VR itself to build and save the global map. When the device enters the space, it will load the global map and reposition at the same time. Successful repositioning means that the device coordinates are integrated into the global map. The virtual game map of the VR large space solution is unified with the global map. After repositioning, each player wearing an AR / VR device and entering the space corresponds to a virtual target, thereby realizing a multi-person immersive interactive experience.

[0004] However, the difficulty of the inside-out large space positioning solution in the prior art lies in how to build an effective global map. Because SLAM is the core underlying algorithm of 6DoF AR / VR, different manufacturers have different degrees of openness. Some devices can provide posture and 3D point cloud data. Some only provide posture and cannot provide 3D point cloud data. Therefore, the solutions for building a global map in the prior art almost all require AR / VR equipment manufacturers to customize the relevant projects. The solution for building a global map can only be applied to specific devices and cannot be universal between different types of AR / VR devices, resulting in higher costs. Summary of the invention

[0005] The present application provides a method and electronic device for constructing a global map, which are used to realize the construction of a global map in a large space, are applicable to various types of AR / VR devices, are universal, and save costs.

[0006] In a first aspect, an embodiment of the present application provides a method for constructing a global map, the method comprising:

[0007] Receive global map building instructions;

[0008] In response to the received global map building instruction, determining the camera poses corresponding to the pre-acquired multiple frames of scene images, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image;

[0009] Extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain respective ORB feature points in the multiple frames of scene images;

[0010] For each frame of the scene image in the multiple frames of scene images, reconstruct the 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image;

[0011] A global map is constructed according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence.

[0012] In the above-mentioned embodiment of the present application, ORB feature extraction is performed on the multiple frames of scene images pre-acquired by the electronic device to obtain each ORB feature point in the multiple frames of scene images; then, the 3D point cloud data of each of the multiple frames of scene images is reconstructed using each ORB feature point in the multiple frames of scene images to obtain the 3D point cloud data corresponding to the multiple frames of scene images. Finally, a global map is constructed using the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images. Therefore, the map construction method in the embodiment of the present application is applicable to any electronic device, regardless of whether the electronic device can obtain point cloud data, so the method in the embodiment of the present application is universal, does not need to be customized, and saves costs.

[0013] In one embodiment, if the electronic device is a single-camera device, the number of images in each frame of scene image is single;

[0014] The reconstructing, for each frame of the multiple frames of scene images, 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image includes:

[0015] For each frame of the multiple frames of scene images, using a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain matched feature point pairs, wherein the 3D feature points corresponding to the two ORB feature points in each feature point pair are the same;

[0016] For each of the feature point pairs, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, obtain the position coordinates of the 3D feature point corresponding to the feature point pair;

[0017] The 3D point cloud data of the scene image is obtained by using the position coordinates of the 3D feature points corresponding to each pair of feature points.

[0018] In the embodiment of the present application, when the electronic device is a single-camera device, the position coordinates of each 3D feature point are determined by the feature point pairs obtained by inter-frame tracking to ensure that 3D point cloud data can be reconstructed.

[0019] In one embodiment, if the electronic device is a multi-camera device, the number of images in each frame of scene image is multiple, and the number of images in each frame of scene image is the same as the number of cameras in the multi-camera device;

[0020] The reconstructing, for each frame of the multiple frames of scene images, 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image includes:

[0021] For each of the multiple scene images, use a tracking algorithm to match the ORB feature points in the multiple scene images in each of the scene images to obtain each matched feature point pair; use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain each matched feature point pair; wherein the multiple scene images correspond to the same number of frames; and for each of the multiple scene images, the camera that photographed the previous frame of the scene image is the same as the camera that photographed the scene image;

[0022] For each of the feature point pairs, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, obtain the position coordinates of the 3D feature point corresponding to the feature point pair;

[0023] The 3D point cloud data of the scene image is derived by using the position coordinates of the 3D feature points corresponding to each feature point pair.

[0024] In the embodiment of the present application, when the electronic device is a multi-camera device, it is necessary to determine the position coordinates of each 3D feature point through feature point pairs obtained by inter-frame tracking and intra-frame tracking to ensure that the reconstructed 3D point cloud data is more accurate.

[0025] In one embodiment, after obtaining the 3D point cloud data of the scene image by using the position coordinates of the 3D feature points corresponding to each feature point pair, the method further includes:

[0026] For each 3D feature point in the 3D point cloud data of the scene image, construct a reprojection error corresponding to the 3D feature point using the 3D feature point and a feature point pair corresponding to the 3D feature point; and optimize the position coordinates of the 3D feature point using the reprojection error to obtain the target position coordinates of the 3D feature point;

[0027] According to the target position coordinates of each 3D feature point in the 3D point cloud data, optimized 3D point cloud data is obtained, and the optimized 3D point cloud data is determined as the 3D point cloud data of the scene image.

[0028] In the embodiment of the present application, the 3D point cloud data corresponding to the multiple frames of scene images are optimized by constructing a reprojection error equation to ensure the accuracy of the determined 3D point cloud data, thereby further improving the accuracy of the constructed global map.

[0029] In one embodiment, the identification of the camera pose, the identification of the ORB feature point, and the identification of the 3D point cloud data corresponding to each frame of the multiple frames of scene images are all the same;

[0030] After constructing the global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, the method further includes:

[0031] When the user's electronic device enters a large space scene, the current frame scene image is obtained at a specified time interval;

[0032] Extracting ORB feature points from the current frame scene image to obtain each ORB feature point in the current frame scene image;

[0033] Searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image;

[0034] If the target ORB feature point is found, a pose conversion relationship is determined using the ORB feature point of the current frame scene image, a camera pose with the same identifier as the target ORB feature point, and 3D point cloud data, wherein the pose conversion relationship is used to convert the camera pose corresponding to the current frame scene image acquired by the electronic device entering the large space scene into the global map;

[0035] The camera posture corresponding to the current frame scene image is transformed by using the posture transformation relationship to obtain a relocated target posture, and the virtual object corresponding to the user is displayed in the large space scene by using the relocated target posture.

[0036] In the embodiment of the present application, when the electronic device enters the large space, the global map is loaded and repositioned at the same time, that is, the coordinates of the electronic device are integrated into the global map. This ensures that the virtual game map in the large space is unified with the global map, ensures that the position of the virtual object of the user corresponding to the electronic device entering the large space is accurate, and improves the user experience.

[0037] In one embodiment, searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image includes:

[0038] For each ORB feature point in the current frame scene image, perform the following steps:

[0039] Obtaining the similarity between the ORB feature point and each ORB feature point in the global map according to the position coordinates of the ORB feature point and the position coordinates of each ORB feature point in the global map;

[0040] Determine an ORB feature point in the global map having the highest similarity value with the ORB feature point as an intermediate ORB feature point corresponding to the ORB feature point;

[0041] If the similarity between the intermediate ORB feature point and the ORB feature point is greater than a specified value, the intermediate ORB feature point is determined as a target ORB feature point of the ORB feature point;

[0042] If the similarity between the intermediate ORB feature point and the ORB feature point is not greater than the specified value, returning to the step of acquiring the current frame scene image every specified time length.

[0043] In the embodiment of the present application, the similarity between each ORB feature point in the current frame scene image and each ORB feature point in the global map is obtained by using the position coordinates of each ORB feature point in the current frame scene image and the position coordinates of each ORB feature point in the global map, and the target ORB feature point is determined by using the similarity between each ORB feature point in the current frame scene image and each ORB feature point in the global map. Thus, the accuracy of the determined target ORB feature point is ensured.

[0044] In one embodiment, the determining of the posture conversion relationship using the ORB feature points of the current frame scene image, the camera posture with the same identifier as the target ORB feature point, and the 3D point cloud data includes:

[0045] The ORB feature points of the current frame scene image, the camera pose with the same identity as the target ORB feature points, and the 3D point cloud data are input into the pnp algorithm to obtain the pose conversion relationship.

[0046] In the embodiment of the present application, the posture conversion relationship is determined by combining the ORB feature points of the previous frame scene image, the camera posture with the same identifier as the target ORB feature point, and the 3D point cloud data. That is, the posture conversion relationship is determined based on the data in the virtual map and the global map data, ensuring the accuracy of the determined posture conversion relationship.

[0047] In one embodiment, before extracting ORB features of the multiple frames of scene images respectively, the method further includes:

[0048] Determining whether the electronic device can acquire 3D point cloud data;

[0049] If yes, determining whether the 3D point cloud data acquired by the electronic device can meet the specified conditions;

[0050] If yes, executing the step of constructing a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images;

[0051] If not, the step of respectively extracting ORB features of the scene images from the multiple frames of scene images is executed; if not, the step of respectively extracting ORB features of the multiple frames of scene images pre-acquired by the electronic device is executed.

[0052] In the embodiment of the present application, before reconstructing the 3D point cloud data, it is first necessary to determine whether the electronic device can obtain the 3D point cloud data and whether the obtained 3D point cloud data can meet the specified conditions. This ensures that only electronic devices that cannot obtain the 3D point cloud data or that can obtain the 3D point cloud data but the 3D point cloud data does not meet the specified conditions need to reconstruct the 3D point cloud data. For electronic devices that can obtain the 3D point cloud data and the 3D point cloud data can meet the specified conditions, reconstruction is not required and the 3D point cloud data can be used directly, thereby saving computing resources and improving the efficiency of building a global map.

[0053] In one embodiment, whether the 3D point cloud data acquired by the electronic device meets the specified condition is determined in the following manner:

[0054] For each frame of the scene image in the multiple frames of scene images, reproject the 3D point cloud data corresponding to the scene image into the scene image using a preset projection transformation matrix to obtain each 2D projection point of the 3D point cloud data in the scene image;

[0055] Performing corner point detection on the 2D projection points to determine whether each 2D projection point in the scene image is a corner point;

[0056] If all projection points in the multiple frames of scene images are corner points, determining that the 3D point cloud data acquired by the electronic device meets the specified condition;

[0057] Otherwise, it is determined that the 3D point cloud data acquired by the electronic device does not meet the specified condition.

[0058] In the embodiment of the present application, quality assessment is performed on the point cloud data acquired by the electronic device to determine whether the quality of the acquired 3D point cloud data meets the requirements, so as to ensure the accuracy of the constructed global map.

[0059] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0060] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0061] Receive global map building instructions;

[0062] In response to the received global map building instruction, determining the camera poses corresponding to the pre-acquired multiple frames of scene images, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image;

[0063] Extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain respective ORB feature points in the multiple frames of scene images;

[0064] For each frame of the scene image in the multiple frames of scene images, reconstruct the 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image;

[0065] A global map is constructed according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence.

[0066] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0068] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0069] Figure 2 One of the flowcharts of a method for constructing a global map in a large space provided in an embodiment of the present application is exemplarily shown;

[0070] Figure 3 The schematic diagram of the process of determining whether 3D point cloud data needs to be reconstructed provided by the embodiment of the present application is exemplarily shown;

[0071] Figure 4 The flowchart for determining whether 3D point cloud data meets specified conditions provided in an embodiment of the present application is exemplarily shown;

[0072] Figure 5 A schematic diagram of a multi-frame scene image provided by an embodiment of the present application is exemplarily shown;

[0073] Figure 6 The schematic diagram of a sliding window provided in an embodiment of the present application is exemplarily shown;

[0074] Figure 7 The schematic diagram of the process of the relocation method in a large space provided by the embodiment of the present application is exemplarily shown;

[0075] Figure 8 The following is a schematic diagram showing a flow chart of a method for searching a target scene image provided in an embodiment of the present application;

[0076] Fig. 9 The second flowchart of the method for constructing a global map in a large space provided by an embodiment of the present application is exemplarily shown;

[0077] Fig.10 A schematic diagram of a device for constructing a global map in a large space provided by an embodiment of the present application is exemplarily shown;

[0078] Fig.11 The structural diagram of the electronic device provided by the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0079] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0080] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0081] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0082] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0083] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0084] The idea of ​​the embodiments of the present application is summarized below.

[0085] At present, the methods of constructing global maps in large-space positioning solutions require AR / VR equipment manufacturers to carry out relevant project customization. The solutions for constructing global maps can only be applied to specific devices and are not universal between different types of AR / VR devices, resulting in high costs.

[0086] Based on the existing technology, there is a problem that the solution for constructing a global map is not universal among AR / VR devices, resulting in high costs. The embodiment of the present application provides a method for constructing a global map in a large space, by performing ORB feature extraction on multiple frames of scene images pre-acquired by an electronic device, and obtaining each ORB feature point in the multiple frames of scene images; then, the 3D point cloud data of each of the multiple frames of scene images is reconstructed using each ORB feature point in the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images are obtained. Finally, a global map is constructed using the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images. Therefore, the map construction method in the embodiment of the present application is applicable to any electronic device, regardless of whether the electronic device can obtain point cloud data, so the method in the embodiment of the present application is universal, does not need to be customized, and saves costs.

[0087] like Figure 1 As shown, an application scenario of a method for constructing a global map in a large space, in which the electronic device is a VR device as an example for explanation in the embodiment of the present application. The application scenario includes a server 110 and a VR device 120.

[0088] In a possible application scenario, any VR device 120 in the application scenario may perform the following steps:

[0089] After receiving the global map construction instruction, the VR device 120 determines the camera poses corresponding to the pre-acquired multiple frames of scene images in response to the received global map construction instruction, wherein the camera pose corresponding to any one of the multiple frames of scene images is acquired and stored by the VR device when the arbitrary frame of scene images is acquired; the VR device 120 performs ORB feature extraction on the multiple frames of scene images to obtain the respective ORB feature points in the multiple frames of scene images. For each frame of the multiple frames of scene images, the VR device 120 reconstructs the 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image; and constructs a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images. The constructed global map is sent to the server 110 so that any VR device 120 among the VR devices can enter the large space scene. The VR device 120 obtains the current frame scene image at a specified time interval; and extracts ORB feature points of the current frame scene image to obtain each ORB feature point in the current frame scene image; then the target ORB feature point is searched for in the global map using the ORB feature point of the current frame scene image; if the target ORB feature point is found, the VR device 120 uses the ORB feature point of the current frame scene image, the camera pose with the same identifier as the target ORB feature point, and the 3D point cloud data to determine the pose conversion relationship, wherein the pose conversion relationship is used to convert the camera pose corresponding to the current frame scene image obtained by the electronic device entering the large space scene into the global map; then the VR device 120 uses the pose conversion relationship to perform pose conversion on the camera pose corresponding to the current frame scene image to obtain the relocated target pose, and uses the relocated target pose to display the virtual object corresponding to the user in the large space scene.

[0090] in, Figure 1 The server 110 and the VR device 120 may exchange information via a communication network, wherein the communication method adopted by the communication network may be divided into a wireless communication method or a wired communication method.

[0091] Exemplarily, the VR device 120 can access the network through cellular mobile communication technology to communicate with the server 110, wherein the cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0092] Optionally, the VR device 120 may access the network and communicate with the server 110 via short-range wireless communication, wherein the short-range wireless communication method, for example, includes Wireless Fidelity (Wi-Fi) technology.

[0093] Among them, the description in this application only details a single server 110 and three VR devices 120, but those skilled in the art should understand that the server 110 and VR device 120 shown are intended to represent the operation of the server 110 and VR device 120 involved in the technical solution of this application. It does not imply any limitation on the number, type or location of the server 110 and VR device 120. It should be noted that if additional modules are added to the illustrated environment or individual modules are removed from it, the underlying concept of the example embodiment of this application will not be changed.

[0094] It should be noted that the method for constructing a global map in a large space proposed in this application is not only applicable to Figure 1 The application scenario shown is also applicable to any device that builds a global map in a large space.

[0095] The following describes a method for constructing a global map in a large space according to an exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of the present application, and the implementation methods of the present application are not subject to any limitations in this regard.

[0096] Before introducing the method for constructing a global map in a large space in this application, the method for constructing a global map in a large space in this application is first introduced. Figure 2 As shown, it is a flow chart of a method for constructing a global map in a large space, which may specifically include the following steps:

[0097] Step 201: receiving a global map construction instruction;

[0098] Step 202: In response to the received global map building instruction, determining the camera poses corresponding to the pre-acquired multiple frames of scene images, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image;

[0099] Step 203: extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain ORB feature points of the multiple frames of scene images;

[0100] In the embodiment of the present application, ORB feature extraction is performed on any frame of scene image using an ORB feature extraction algorithm.

[0101] In order to improve the efficiency of building a global map, in the embodiment of the present application, only when the electronic device cannot obtain the 3D point cloud data or the 3D point cloud data obtained by the electronic device cannot meet the specified conditions, it is necessary to reconstruct the 3D point cloud data. In one embodiment, before executing step 203, Figure 3 As shown, it is a schematic diagram of a process for determining whether 3D point cloud data needs to be reconstructed, which may specifically include the following steps:

[0102] Step 301: Determine whether the electronic device can obtain 3D point cloud data, if yes, execute step 302, if no, execute step 304;

[0103] In one embodiment, whether the electronic device can acquire 3D point cloud data may be determined in the following manner:

[0104] Using the pre-set correspondence between the electronic device identifier and the 3D point cloud identifier, determine the target 3D point cloud identifier corresponding to the electronic device identifier; if the target 3D point cloud identifier is the first designated identifier, determine that the electronic device can obtain 3D point cloud data; if the target 3D point cloud data identifier is the second designated identifier, determine that the electronic device cannot obtain 3D point cloud data; wherein the 3D point cloud identifier is used to indicate whether the 3D point cloud data can be obtained.

[0105] The first designated identifier in the embodiment of the present application may be 1, and the second designated identifier may be 0, but the embodiment of the present application does not limit the first designated identifier and the second designated identifier. The specific values ​​of the first designated identifier and the second designated identifier in the embodiment of the present application may be set according to the specific actual situation.

[0106] Step 302: determining whether the 3D point cloud data acquired by the electronic device can meet the specified conditions, if yes, executing step 303, if no, executing step 304;

[0107] Next, the determination of whether the 3D point cloud data acquired by the electronic device meets the specified conditions in the embodiment of the present application is described. Figure 4 As shown, it is a schematic diagram of a process for determining whether 3D point cloud data meets specified conditions, which may specifically include the following steps:

[0108] Step 3021: for any one of the pre-acquired multiple scene image frames, reproject the 3D point cloud data corresponding to the any one of the scene image frames into the any one of the scene image frames using a preset projection transformation matrix to obtain 2D projection points of the 3D point cloud data in the any one of the scene image frames;

[0109] In the embodiment of the present application, multiple frames of scene images at various angles and locations are pre-acquired to ensure that the global map is constructed more accurately. However, the acquired multiple frames of scene images are not limited in the embodiment of the present application, and the number of frames of the scene images acquired in the embodiment of the present application can be set according to the actual situation.

[0110] The scene image projection method in the embodiment of the present application does not belong to the invention point in the present application, so the implementation of the present application will not be described in detail here.

[0111] In one embodiment, step 3021 can be specifically implemented as follows: for any 3D point in the 3D point cloud data corresponding to any frame of scene image, multiply the three-dimensional position coordinates of the 3D point by a preset projection transformation matrix to obtain the two-dimensional position coordinates of the 2D projection point corresponding to the any 3D point.

[0112] Step 3022: performing corner point detection on the 2D projection points to determine whether all 2D projection points in any frame of scene image are corner points; if so, executing step 3023; if not, executing step 3024;

[0113] Step 3023: Determine whether the 3D point cloud data acquired by the electronic device meets the specified condition;

[0114] Step 3024: Determine whether the 3D point cloud data acquired by the electronic device satisfies the specified condition.

[0115] It should be noted that the corner point detection method in the embodiment of the present application may be the ORB corner point detection method. The embodiment of the present application does not limit the corner point detection method. The corner point detection method in the embodiment of the present application may be set according to actual conditions.

[0116] Therefore, in the embodiment of the present application, by performing a quality assessment on the point cloud data acquired by the electronic device, it is determined that the quality of the acquired 3D point cloud data meets the requirements, so as to ensure the accuracy of the constructed global map.

[0117] Step 303: constructing a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images (corresponding to step 205);

[0118] Step 304: extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain ORB feature points of each of the multiple frames of scene images (corresponding to step 203).

[0119] In the embodiment of the present application, only when the electronic device cannot obtain 3D point cloud data or the electronic device can obtain 3D point cloud data but the obtained 3D point cloud data does not meet the specified conditions, it is necessary to perform steps 203 to 204 to reconstruct the 3D point cloud data. If the electronic device can obtain 3D point cloud data and the obtained 3D point cloud data meets the specified conditions, step 205 can be directly executed to build a global map.

[0120] In one embodiment, whether the electronic device can acquire 3D point cloud data may be determined in the following manner:

[0121] Utilizing the pre-set correspondence between the device identifier and the acquisition identifier, determine the target acquisition identifier corresponding to the device identifier of the electronic device; if the target acquisition identifier is the third designated identifier, determine that the electronic device can acquire 3D point cloud data; if the target acquisition identifier is the fourth designated identifier, determine that the electronic device cannot acquire the 3D point cloud.

[0122] It should be noted that the third designated identifier and the fourth designated identifier in the embodiment of the present application can be set according to actual conditions, and the third designated identifier and the fourth designated identifier in the embodiment of the present application are not limited here.

[0123] Step 204: for each frame of the scene image in the multiple frames of scene images, reconstruct the 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image;

[0124] In one embodiment, step 204 can be specifically implemented in the following two ways:

[0125] Method 1: If the electronic device is a single-camera device, for each frame of the multiple scene images, use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain matched feature point pairs, wherein the 3D feature points corresponding to the two ORB feature points in each feature point pair are the same; for each feature point pair in the feature point pairs, obtain the position coordinates of the 3D feature points corresponding to the feature point pair based on the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are respectively located; and use the position coordinates of the 3D feature points corresponding to each feature point pair to obtain 3D point cloud data of the scene image.

[0126] The number of images in each frame of scene image acquired by the single camera device in the embodiment of the present application is single.

[0127] Therefore, in the embodiment of the present application, when the electronic device is a single-camera device, the position coordinates of each 3D feature point are determined by the feature point pairs obtained by inter-frame tracking to ensure that 3D point cloud data can be reconstructed.

[0128] Method 2: If the electronic device is a multi-camera device, for each frame of the multiple scene images, use a tracking algorithm to match the ORB feature points in the multiple scene images in each frame of the scene image to obtain matched feature point pairs; use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain matched feature point pairs; wherein the multiple scene images correspond to the same number of frames; and for each scene image in the multiple scene images, the camera that captures the previous frame of the scene image is the same as the camera that captures the scene image; for each feature point pair in each feature point pair, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are respectively located, obtain the position coordinates of the 3D feature points corresponding to the feature point pair; and use the position coordinates of the 3D feature points corresponding to each feature point pair to obtain 3D point cloud data of the scene image.

[0129] The multi-camera device in the embodiment of the present application acquires multiple images for each frame of scene images, and the number of images for each frame of scene images is the same as the number of cameras in the multi-camera device.

[0130] Therefore, in the embodiment of the present application, when the electronic device is a multi-camera device, it is necessary to determine the position coordinates of each 3D feature point through feature point pairs obtained by inter-frame tracking and intra-frame tracking to ensure that the reconstructed 3D point cloud data is more accurate.

[0131] In the embodiment of the present application, a specific method for determining a feature point pair of any two scene images is as follows: using the KLT algorithm to track feature points of the any two scene images to obtain an initial feature point pair of the any two scene images, using the ORB feature extraction algorithm to perform ORB feature extraction on two feature points in the initial feature point pair respectively to obtain the ORB features of each of the two feature points, and obtaining the similarity between the two feature points through the ORB features of each of the two feature points. If the similarity is greater than a specified value, the initial feature point pair is determined to be the matched feature point pair; if the similarity is not greater than the specified value, the initial feature point pair is deleted.

[0132] like Figure 5 As shown, the scene image is described by taking a multi-camera device as an example. And in the embodiment of the present application, a dual camera is taken as an example. Figure 5It can be seen that each frame corresponds to two scene images, and 3D point 1 is captured in scene image a in the previous frame and scene image a in the current frame. 3D point 2 is captured in scene image a and scene image b in the current frame. Therefore, it is necessary to perform feature point matching between the previous frame and the current frame, and it is necessary to perform feature point matching within the frame for scene image a and scene image b in each frame.

[0133] The KLT algorithm and ORB feature extraction algorithm used in the embodiments of the present application belong to the prior art and are not the invention of the present application. The embodiments of the present application will not be described in detail here.

[0134] In the embodiment of the present application, depth restoration is performed by triangulation based on the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, so as to obtain the position coordinates of the 3D feature points corresponding to the 2D feature point pair.

[0135] In the embodiment of the present application, the depth restoration by triangulation to determine the position coordinates of the 3D point belongs to the prior art, and the embodiment of the present application will not be described in detail here.

[0136] In order to ensure that the determined 3D point cloud data is more accurate and further improve the accuracy of the constructed global map, in one embodiment, after executing step 204, for each 3D feature point in the 3D point cloud data of the scene image, the 3D feature point and the feature point pair corresponding to the 3D feature point are used to construct the reprojection error corresponding to the 3D feature point; and the position coordinates of the 3D feature point are optimized using the reprojection error to obtain the target position coordinates of the 3D feature point; based on the target position coordinates of each 3D feature point in the 3D point cloud data, optimized 3D point cloud data are obtained, and the optimized 3D point cloud data are determined as the 3D point cloud data of the scene image.

[0137] In one embodiment, the position coordinates of the 3D feature points can be optimized in the following manner: for each 3D feature point in the 3D point cloud data of the scene image, the 3D feature point is reprojected to the corresponding target scene image using the position coordinates of the 3D feature point and the camera pose of the target scene image to obtain the projection position coordinates of the 3D feature point; wherein the target scene image is a scene image where any ORB feature point corresponding to the 3D feature point is located; using the projection position coordinates of the 3D feature point and the position coordinates of any ORB feature point, the projection position coordinates of the 3D feature point are reprojected to the corresponding target scene image using the position coordinates of the 3D feature point and the camera pose of the target scene image to obtain the projection position coordinates of the 3D feature point; The reprojection error is constructed by using the position coordinates of the 3D feature point and any of the ORB feature points. If the reprojection error is not less than the reprojection error obtained last time, or the reprojection error is greater than a specified threshold, the position coordinates of the 3D feature point are adjusted, and the process returns to the step of reprojecting the 3D feature point into the corresponding target scene image using the position coordinates of the 3D feature point and the camera pose of the target scene image, until the reprojection error is less than the reprojection error obtained last time and the reprojection error is not greater than the specified threshold, and the position coordinates of the adjusted 3D feature point are determined as the target position coordinates of the 3D feature point.

[0138] The method of reprojecting 3D feature points using camera posture in the embodiment of the present application does not belong to the invention point in the present application, and the embodiment of the present application will not be described in detail here.

[0139] Therefore, in the embodiment of the present application, the 3D point cloud data corresponding to multiple frames of scene images are optimized by constructing a reprojection error equation to ensure the accuracy of the determined 3D point cloud data, thereby further improving the accuracy of the constructed global map.

[0140] In the embodiment of the present application, the position coordinates of the 3D feature points can be optimized by means of a sliding window, and the sliding window follows the first-in-first-out principle. Figure 6 , which is a schematic diagram of a sliding window. The length of the sliding window is fixed. Every time a new scene image frame enters the sliding window, an old scene image frame slides out from the bottom of the sliding window in sequence.

[0141] Step 205: Construct a global map based on the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence.

[0142] The global map in the embodiment of the present application includes the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images. In the embodiment of the present application, the camera poses, ORB feature points, and 3D point cloud data all have corresponding identifiers, and the camera poses, ORB feature points, and 3D point cloud data corresponding to the same frame of scene images are all the same. And the global map in the embodiment of the present application is stored through a feature dictionary, and a feature index table is established through MILD (MULTI-INDEX HASHING FOR, building a dictionary based on multi-index hashing). To speed up 2D feature matching and improve the efficiency of subsequent relocation.

[0143] After introducing the method for constructing the complete local map, the method for relocating the large space in the embodiment of the present application is introduced. In one embodiment, after executing step 205, it is necessary to relocate the electronic device entering the large space, such as Figure 7 FIG. 1 is a flow chart of a relocation method in a large space, which may specifically include the following steps:

[0144] Step 701: When a user's electronic device enters a large space scene, a current frame scene image is obtained at a specified time interval; in the embodiment of the present application, each user's electronic device entering the large space scene needs to execute steps 701 to 705.

[0145] Furthermore, the specified duration in the application embodiment can be set according to specific actual conditions, and the embodiment of the present application does not limit the specific value of the specified duration.

[0146] Step 702: performing ORB feature extraction on the current frame scene image to obtain each ORB feature point in the current frame scene image;

[0147] In the embodiment of the present application, an ORB feature extraction algorithm is used to perform ORB feature extraction on the current frame scene image to obtain various ORB feature points.

[0148] Step 703: searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image;

[0149] Next, the method of searching for the target scene image in the embodiment of the present application is introduced. Figure 8 As shown, it is a schematic diagram of the process of searching for a target scene image, which may specifically include the following steps:

[0150] Step 801: for any ORB feature point in the current frame scene image, obtain the similarity between the ORB feature point and each ORB feature point in the global map according to the position coordinates of the ORB feature point and the position coordinates of each ORB feature point in the global map;

[0151] Among them, the similarity between two ORB feature points can be obtained by formula (1):

[0152]

[0153] Among them, K is the similarity between two ORB feature points, x1 is the horizontal position coordinate of an ORB feature point, y1 is the vertical position coordinate of an ORB feature point, x2 is the horizontal position coordinate of another ORB feature point, and y2 is the vertical position coordinate of another ORB feature point.

[0154] Step 802: determining the ORB feature point in the global map having the highest similarity value with the ORB feature point as the middle ORB feature point corresponding to the ORB feature point;

[0155] Step 803: Determine whether the similarity between the intermediate ORB feature point and any one of the ORB feature points is greater than a specified value, if so, execute step 804, if not, return to execute step 701;

[0156] Step 804: Determine the intermediate ORB feature point as a target ORB feature point of any one of the ORB feature points.

[0157] The specified numerical values ​​in the embodiments of the present application can be set according to actual conditions, and the embodiments of the present application do not limit the specified numerical values.

[0158] Thus, the target ORB feature point is determined by the similarity between each ORB feature point in the current frame scene image and each ORB feature point in the global map, thereby ensuring the accuracy of the determined target ORB feature point.

[0159] Step 704: if the target ORB feature point is found, determine a pose conversion relationship using the ORB feature point of the current frame scene image, the camera pose with the same identifier as the target ORB feature point, and the 3D point cloud data, wherein the pose conversion relationship is used to convert the camera pose corresponding to the current frame scene image acquired by the electronic device entering the large space scene into the global map;

[0160] In one embodiment, step 704 can be specifically implemented as follows: inputting the ORB feature points of the current frame scene image, the camera pose with the same identity as the target ORB feature points, and the 3D point cloud data into the pnp algorithm to derive the pose conversion relationship.

[0161] In an embodiment of the present application, if the current frame scene image includes multiple ORB feature points, each group of ORB feature point data (ORB feature points of the current frame scene image, camera poses with the same identifier as the target ORB feature points, and 3D point cloud data) corresponds to a pose transformation relationship, and the mean of each pose transformation relationship is determined as the final pose transformation relationship.

[0162] The pnp algorithm in the embodiment of the present application will also output the number of inner points corresponding to the pose transformation relationship. In order to ensure the accuracy of the relocated target pose, in one embodiment, before executing step 705, it is determined that the number of inner points is greater than the total number of ORB feature points in the current frame scene image.

[0163] In one embodiment, if it is determined that the number of inliers is not greater than the total number of ORB feature points of the current frame scene image, the process returns to step 701 .

[0164] To enhance the accuracy of relocalization, specific landmark positioning points can be posted in the large space. If there is a specific landmark in the current frame scene image, the PNP calculation of the specific landmark is used to optimize the pose transformation relationship.

[0165] Step 705: Perform posture transformation on the camera posture corresponding to the current frame scene image using the posture transformation relationship to obtain a relocated target posture, and use the relocated target posture to display the virtual object corresponding to the user in the large space scene.

[0166] In one embodiment, step 705 may be specifically implemented as follows: multiplying the camera pose corresponding to the current frame scene image by the matrix corresponding to the pose conversion relationship to obtain the relocated target pose.

[0167] Therefore, in the embodiment of the present application, when the electronic device enters the large space, the global map will be loaded and repositioned at the same time, that is, the coordinates of the electronic device will be integrated into the global map. This ensures that the virtual game map in the large space is unified with the global map, ensures that the position of the virtual object of the user corresponding to the electronic device entering the large space is accurate, and improves the user experience.

[0168] In order to further understand the method of constructing a global map in a large space in the embodiment of the present application, as shown in Fig. 9As shown, it is a flow chart of a method for constructing a global map in a large space, which may specifically include the following steps:

[0169] Step 901: receiving a global map construction instruction;

[0170] Step 902: In response to the received global map building instruction, determining the camera poses corresponding to the pre-acquired multiple frames of scene images, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image;

[0171] Step 903: extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain ORB feature points of the multiple frames of scene images;

[0172] Step 904: if the electronic device is a single-camera device, for each frame of the multiple frames of scene images, use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain matched feature point pairs, wherein the 3D feature points corresponding to the two ORB feature points in each feature point pair are the same;

[0173] Step 905: If the electronic device is a multi-camera device, for each frame of the multiple scene images, use a tracking algorithm to match the ORB feature points in the multiple scene images in each frame of the scene image to obtain each matched feature point pair; use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain each matched feature point pair; wherein the multiple scene images correspond to the same number of frames; and for each scene image in the multiple scene images, the camera that captured the previous frame of the scene image is the same as the camera that captured the scene image;

[0174] Step 906: for each of the feature point pairs, obtaining the position coordinates of the 3D feature points corresponding to the feature point pair according to the position coordinates of the two ORB feature points in the feature point pair and the camera poses of the scene images where the two ORB feature points are located;

[0175] Step 907: Obtain 3D point cloud data of the scene image using the position coordinates of the 3D feature points corresponding to each feature point pair;

[0176] Step 908: constructing a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of the 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence;

[0177] Step 909: when the user's electronic device enters the large space scene, the current frame scene image is obtained at a specified time interval;

[0178] Step 910: extracting ORB feature points from the current frame scene image to obtain each ORB feature point in the current frame scene image;

[0179] Step 911: searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image;

[0180] Step 912: If the target ORB feature point is found, determine the pose conversion relationship using the ORB feature point of the current frame scene image, the camera pose with the same identifier as the target ORB feature point, and the 3D point cloud data, wherein the pose conversion relationship is used to convert the camera pose corresponding to the current frame scene image obtained by the electronic device entering the large space scene into the global map;

[0181] Step 913: Perform posture transformation on the camera posture corresponding to the current frame scene image using the posture transformation relationship to obtain a relocated target posture, and use the relocated target posture to display the virtual object corresponding to the user in the large space scene.

[0182] Based on the same inventive concept, the method for constructing a global map in a large space as described above can also be implemented by a device for constructing a global map in a large space. The effect of the device for constructing a global map in a large space is similar to that of the aforementioned method, and will not be repeated here.

[0183] Fig.10 The present invention is a schematic diagram of the structure of an apparatus for constructing a global map in a large space according to an embodiment of the present invention.

[0184] like Fig.10 As shown, the device 1000 for constructing a global map in a large space of the present disclosure may include a receiving module 1010 , a camera pose determination module 1020 , an ORB feature extraction module 1030 , a 3D feature point reconstruction module 1040 and a global map construction module 1050 .

[0185] A receiving module 1010 is used to receive a global map building instruction;

[0186] The camera pose determination module 1020 is used to determine the camera poses corresponding to the pre-acquired multiple frames of scene images in response to the received global map construction instruction, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image;

[0187] The ORB feature extraction module 1030 is used to extract the ORB features of the scene images respectively from the multiple frames of scene images to obtain the ORB feature points of each of the multiple frames of scene images;

[0188] A 3D feature point reconstruction module 1040 is used to reconstruct the 3D point cloud data of each frame of the scene image in the multiple frames of scene images by using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image;

[0189] The global map construction module 1050 is used to construct a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence.

[0190] In one embodiment, if the electronic device is a single-camera device, the number of images in each frame of scene image is single;

[0191] The 3D feature point reconstruction module 1040 is specifically used for:

[0192] For each frame of the multiple frames of scene images, using a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain matched feature point pairs, wherein the 3D feature points corresponding to the two ORB feature points in each feature point pair are the same;

[0193] For each of the feature point pairs, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, obtain the position coordinates of the 3D feature point corresponding to the feature point pair;

[0194] The 3D point cloud data of the scene image is derived by using the position coordinates of the 3D feature points corresponding to each feature point pair.

[0195] In one embodiment, if the electronic device is a multi-camera device, the number of images in each frame of scene image is multiple, and the number of images in each frame of scene image is the same as the number of cameras in the multi-camera device;

[0196] The 3D feature point reconstruction module 1040 is specifically used for:

[0197] For each of the multiple scene images, use a tracking algorithm to match the ORB feature points in the multiple scene images in each of the scene images to obtain each matched feature point pair; use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain each matched feature point pair; wherein the multiple scene images correspond to the same number of frames; and for each of the multiple scene images, the camera that photographed the previous frame of the scene image is the same as the camera that photographed the scene image;

[0198] For each of the feature point pairs, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, obtain the position coordinates of the 3D feature point corresponding to the feature point pair;

[0199] The 3D point cloud data of the scene image is derived by using the position coordinates of the 3D feature points corresponding to each feature point pair.

[0200] In one embodiment, the apparatus further comprises:

[0201] The optimization module 1060 is used to obtain the 3D point cloud data of the scene image by using the position coordinates of the 3D feature points corresponding to each feature point pair.

[0202] For each 3D feature point in the 3D point cloud data of the scene image, construct a reprojection error corresponding to the 3D feature point using the 3D feature point and a feature point pair corresponding to the 3D feature point; and optimize the position coordinates of the 3D feature point using the reprojection error to obtain the target position coordinates of the 3D feature point;

[0203] According to the target position coordinates of each 3D feature point in the 3D point cloud data, optimized 3D point cloud data is obtained, and the optimized 3D point cloud data is determined as the 3D point cloud data of the scene image.

[0204] In one embodiment, the identification of the camera pose, the identification of the ORB feature point, and the identification of the 3D point cloud data corresponding to each frame of the multiple frames of scene images are all the same;

[0205] The relocation module 1070 is used to obtain the current frame scene image at a specified time interval when the user's electronic device enters the large space scene;

[0206] Extracting ORB feature points from the current frame scene image to obtain each ORB feature point in the current frame scene image;

[0207] Searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image;

[0208] If the target ORB feature point is found, a pose conversion relationship is determined using the ORB feature point of the current frame scene image, a camera pose with the same identifier as the target ORB feature point, and 3D point cloud data, wherein the pose conversion relationship is used to convert the camera pose corresponding to the current frame scene image acquired by the electronic device entering the large space scene into the global map;

[0209] The camera posture corresponding to the current frame scene image is transformed by using the posture transformation relationship to obtain a relocated target posture, and the virtual object corresponding to the user is displayed in the large space scene by using the relocated target posture.

[0210] In one embodiment, the relocation module 1050 is further configured to:

[0211] For each ORB feature point in the current frame scene image, perform the following steps:

[0212] Obtaining the similarity between the ORB feature point and each ORB feature point in the global map according to the position coordinates of the ORB feature point and the position coordinates of each ORB feature point in the global map;

[0213] Determine an ORB feature point in the global map having the highest similarity value with the ORB feature point as an intermediate ORB feature point corresponding to the ORB feature point;

[0214] If the similarity between the intermediate ORB feature point and the ORB feature point is greater than a specified value, the intermediate ORB feature point is determined as a target ORB feature point of the ORB feature point;

[0215] If the similarity between the intermediate ORB feature point and the ORB feature point is not greater than the specified value, returning to the step of acquiring the current frame scene image every specified time length.

[0216] In one embodiment, the relocation module 1050 is further configured to:

[0217] The ORB feature points of the current frame scene image, the camera pose with the same identity as the target ORB feature points, and the 3D point cloud data are input into the pnp algorithm to obtain the pose conversion relationship.

[0218] In one embodiment, the apparatus further comprises:

[0219] A first judgment module 1080 is used to judge whether the electronic device can acquire 3D point cloud data before performing ORB feature extraction on the multiple frames of scene images respectively;

[0220] If yes, determining whether the 3D point cloud data acquired by the electronic device can meet the specified conditions;

[0221] If yes, executing the step of constructing a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images;

[0222] If not, the step of respectively extracting ORB features of the scene images from the multiple frames of scene images is executed; if not, the step of respectively extracting ORB features of the multiple frames of scene images pre-acquired by the electronic device is executed.

[0223] In one embodiment, the apparatus further comprises:

[0224] The second judgment module 1090 is used to determine whether the 3D point cloud data acquired by the electronic device meets the specified conditions in the following manner:

[0225] For each frame of the scene image in the multiple frames of scene images, reproject the 3D point cloud data corresponding to the scene image into the scene image using a preset projection transformation matrix to obtain each 2D projection point of the 3D point cloud data in the scene image;

[0226] Performing corner point detection on the 2D projection points to determine whether each 2D projection point in the scene image is a corner point;

[0227] If all projection points in the multiple frames of scene images are corner points, determining that the 3D point cloud data acquired by the electronic device meets the specified condition;

[0228] Otherwise, it is determined that the 3D point cloud data acquired by the electronic device does not meet the specified condition.

[0229] After introducing a method and apparatus for constructing a global map in a large space according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.

[0230] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0231] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for constructing a global map in a large space according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201-205 shown in .

[0232] Refer to the following Fig.11 The electronic device 1100 according to this embodiment of the present invention is described. Fig.11 The electronic device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0233] like Fig.11 As shown, the components of the electronic device 1100 may include but are not limited to: the above-mentioned at least one processor 1101, the above-mentioned at least one computer storage medium 1102, and a bus 1103 connecting different system components (including the computer storage medium 1102 and the processor 1101).

[0234] Bus 1103 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0235] Computer storage media 1102 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1121 and / or cache storage media 1122 , and may further include read-only computer storage media (ROM) 1123 .

[0236] The computer storage medium 1102 may also include a program / utility 1125 having a set (at least one) of program modules 1124, such program modules 1124 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0237] The electronic device 1100 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other AR devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1105. In addition, the electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1106. As shown, the network adapter 1106 communicates with other modules for the electronic device 1100 through a bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0238] In some possible implementations, various aspects of the method for constructing a global map in a large space provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the method for constructing a global map in a large space according to various exemplary embodiments of the present invention described above in this specification.

[0239] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for constructing a global map, characterized in that: The method comprises: Receive global map building instructions; In response to the received global map building instruction, determining the camera poses corresponding to the pre-acquired multiple frames of scene images, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image; Extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain respective ORB feature points in the multiple frames of scene images; For each frame of the scene image in the multiple frames of scene images, reconstruct the 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image; A global map is constructed according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence.

2. The method according to claim 1, characterized in that: If the electronic device is a single-camera device, the number of images in each frame of the scene image is single; The reconstructing, for each frame of the multiple frames of scene images, 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image includes: For each frame of the multiple frames of scene images, using a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain matched feature point pairs, wherein the 3D feature points corresponding to the two ORB feature points in each feature point pair are the same; For each of the feature point pairs, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, obtain the position coordinates of the 3D feature point corresponding to the feature point pair; The 3D point cloud data of the scene image is derived by using the position coordinates of the 3D feature points corresponding to each feature point pair.

3. The method according to claim 1, characterized in that If the electronic device is a multi-camera device, the number of images in each frame of scene image is multiple, and the number of images in each frame of scene image is the same as the number of cameras in the multi-camera device; The reconstructing, for each frame of the multiple frames of scene images, 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image includes: For each of the multiple scene images, use a tracking algorithm to match the ORB feature points in the multiple scene images in each of the scene images to obtain each matched feature point pair; use a tracking algorithm to match the ORB feature points in the scene image and the ORB feature points in the previous frame of the scene image to obtain each matched feature point pair; wherein the multiple scene images correspond to the same number of frames; and for each of the multiple scene images, the camera that photographed the previous frame of the scene image is the same as the camera that photographed the scene image; For each of the feature point pairs, according to the position coordinates of the two ORB feature points in the feature point pair and the camera pose of the scene image where the two ORB feature points are located, obtain the position coordinates of the 3D feature point corresponding to the feature point pair; The 3D point cloud data of the scene image is derived by using the position coordinates of the 3D feature points corresponding to each feature point pair.

4. The method according to claim 2 or 3, characterized in that: After obtaining the 3D point cloud data of the scene image by using the position coordinates of the 3D feature points corresponding to each pair of feature points, the method further includes: For each 3D feature point in the 3D point cloud data of the scene image, construct a reprojection error corresponding to the 3D feature point using the 3D feature point and a feature point pair corresponding to the 3D feature point; and optimize the position coordinates of the 3D feature point using the reprojection error to obtain the target position coordinates of the 3D feature point; According to the target position coordinates of each 3D feature point in the 3D point cloud data, optimized 3D point cloud data is obtained, and the optimized 3D point cloud data is determined as the 3D point cloud data of the scene image.

5. The method according to claim 1, characterized in that The identification of the camera pose, the identification of the ORB feature points and the identification of the 3D point cloud data corresponding to each frame of the multiple frames of scene images are all the same; After constructing the global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, the method further includes: When the user's electronic device enters a large space scene, the current frame scene image is obtained at a specified time interval; Extracting ORB feature points from the current frame scene image to obtain each ORB feature point in the current frame scene image; Searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image; If the target ORB feature point is found, a pose conversion relationship is determined using the ORB feature point of the current frame scene image, a camera pose with the same identifier as the target ORB feature point, and 3D point cloud data, wherein the pose conversion relationship is used to convert the camera pose corresponding to the current frame scene image acquired by the electronic device entering the large space scene into the global map; The camera posture corresponding to the current frame scene image is transformed by using the posture transformation relationship to obtain a relocated target posture, and the virtual object corresponding to the user is displayed in the large space scene by using the relocated target posture.

6. The method according to claim 5, characterized in that The step of searching for target ORB feature points in the global map using the ORB feature points of the current frame scene image includes: For each ORB feature point in the current frame scene image, perform the following steps: Obtaining the similarity between the ORB feature point and each ORB feature point in the global map according to the position coordinates of the ORB feature point and the position coordinates of each ORB feature point in the global map; Determine an ORB feature point in the global map having the highest similarity value with the ORB feature point as an intermediate ORB feature point corresponding to the ORB feature point; If the similarity between the intermediate ORB feature point and the ORB feature point is greater than a specified value, the intermediate ORB feature point is determined as a target ORB feature point of the ORB feature point; If the similarity between the intermediate ORB feature point and the ORB feature point is not greater than the specified value, returning to the step of acquiring the current frame scene image every specified time length.

7. The method according to claim 5, characterized in that The determining of the posture conversion relationship by using the ORB feature points of the current frame scene image, the camera posture with the same identifier as the target ORB feature point, and the 3D point cloud data includes: The ORB feature points of the current frame scene image, the camera pose with the same identity as the target ORB feature points, and the 3D point cloud data are input into the pnp algorithm to obtain the pose conversion relationship.

8. The method according to claim 1, characterized in that Before respectively extracting ORB features of the scene images from the multiple frames of scene images, the method further includes: Determining whether the electronic device can acquire 3D point cloud data; If yes, determining whether the 3D point cloud data acquired by the electronic device can meet the specified conditions; If yes, executing the step of constructing a global map according to the camera poses corresponding to the multiple frames of scene images, the ORB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images; If not, executing the step of extracting ORB features of the scene images from the multiple frames of scene images respectively; If not, the step of respectively performing ORB feature extraction on multiple frames of scene images pre-acquired by the electronic device is executed.

9. The method according to claim 8, characterized in that Determine whether the 3D point cloud data acquired by the electronic device meets the specified conditions in the following manner: For each frame of the scene image in the multiple frames of scene images, reproject the 3D point cloud data corresponding to the scene image into the scene image using a preset projection transformation matrix to obtain each 2D projection point of the 3D point cloud data in the scene image; Performing corner point detection on the 2D projection points to determine whether each 2D projection point in the scene image is a corner point; If all projection points in the multiple frames of scene images are corner points, determining that the 3D point cloud data acquired by the electronic device meets the specified condition; Otherwise, it is determined that the 3D point cloud data acquired by the electronic device does not meet the specified condition.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Receive global map building instructions; In response to the received global map building instruction, determining the camera poses corresponding to the pre-acquired multiple frames of scene images, wherein the camera pose corresponding to any one frame of scene image in the multiple frames of scene is acquired and stored by the electronic device when acquiring the any one frame of scene image; Extracting ORB features of the scene images from the multiple frames of scene images respectively to obtain respective ORB feature points in the multiple frames of scene images; For each frame of the scene image in the multiple frames of scene images, reconstruct the 3D point cloud data of the scene image using the ORB feature points of the scene image to obtain the 3D point cloud data of the scene image; A global map is constructed according to the camera poses corresponding to the multiple frames of scene images, the oRB feature points corresponding to the multiple frames of scene images, and the 3D point cloud data corresponding to the multiple frames of scene images, wherein the point cloud data corresponding to the multiple frames of scene images are composed of 3D point cloud data of each frame of scene image, and the scene image and the 3D point cloud data are in one-to-one correspondence.

Images