Positioning system, method, apparatus, and computing device for three-dimensional spatial map

Abstract

Embodiments of the present specification provide a positioning system, method, device and computing equipment of a three-dimensional space map, wherein the positioning system of the three-dimensional space map comprises a collection device, a user device and a server; the collection device is configured to collect environment basic data information of a target environment, and send the environment basic data information to the server, wherein the environment basic data information comprises multi-dimensional sensing data information; the server is configured to receive the environment basic data information, and generate a three-dimensional space map of the target environment according to the environment basic data information; the user device is configured to obtain multimedia data of the target environment, and send the multimedia data to the server based on an extended reality framework; and the server is further configured to determine position information of a user based on the three-dimensional space map and the multimedia data, and send the position information to the user device.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a positioning system for three-dimensional spatial maps. Background Technology

[0002] In practical applications, people sometimes struggle to find their way or to unfamiliar locations. However, with advancements in science and technology, AR real-time navigation has become widely used and popular. However, existing AR navigation technologies are mostly based on Wi-Fi, Bluetooth, or infrared positioning, which are susceptible to interference from other signals, resulting in poor stability and affecting accuracy.

[0003] For indoor positioning technology, Bluetooth-based positioning technology is mainly used for small-scale positioning, with a short transmission distance, resulting in extremely high deployment and maintenance costs in large indoor environments. Infrared-based positioning technology, although it has relatively high indoor positioning accuracy, is susceptible to interference from other lights because light cannot penetrate obstacles and infrared rays can only travel at line of sight. Furthermore, the short transmission distance of infrared light makes indoor positioning less effective. When the device is placed in a pocket or blocked by a wall, it cannot work properly. Antennas need to be installed in each room, so the overall cost of deployment in large spaces will be very high.

[0004] Therefore, there is an urgent need for a method to solve the technical problems of low positioning accuracy and high cost of the above-mentioned positioning technologies. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a positioning system for three-dimensional spatial maps. One or more embodiments of this specification also relate to positioning methods and apparatus for three-dimensional spatial maps, a computing device, an augmented reality (AR) device or a virtual reality (VR) device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a positioning system for a three-dimensional spatial map is provided, comprising: a data acquisition device, a user device, and a server;

[0007] The acquisition device is configured to acquire basic environmental data information of the target environment and send the basic environmental data information to the server based on the extended reality framework, wherein the basic environmental data information includes multi-dimensional sensor data information.

[0008] The server is configured to receive the environmental basic data information and generate a three-dimensional spatial map of the target environment based on the environmental basic data information.

[0009] The user equipment is configured to acquire multimedia data of the target environment and send the multimedia data to the server based on the extended reality framework;

[0010] The server is also configured to determine the user's location information based on the three-dimensional spatial map, the multimedia data, and the visual positioning service, and to send the location information to the user device.

[0011] According to a second aspect of the embodiments of this specification, a method for locating a three-dimensional spatial map is provided, applied to a server, comprising:

[0012] Receive basic environmental data information uploaded by the acquisition device based on the extended reality framework, and generate a three-dimensional spatial map of the target environment based on the basic environmental data information;

[0013] Receive multimedia data of the target environment uploaded by the user equipment based on the extended reality framework;

[0014] The user's location information is determined based on the three-dimensional spatial map, the multimedia data, and the visual positioning service, and the location information is sent to the user device.

[0015] According to a third aspect of the embodiments of this specification, a positioning device for a three-dimensional spatial map is provided, applied to a server, comprising:

[0016] The generation module is configured to receive environmental basic data information uploaded by the acquisition device based on the extended reality framework, and generate a three-dimensional spatial map of the target environment based on the environmental basic data information.

[0017] The first receiving module is configured to receive multimedia data of the target environment uploaded by the user equipment based on the extended reality framework;

[0018] The positioning module is configured to determine the user's location information based on the three-dimensional spatial map, the multimedia data, and the visual positioning service, and send the location information to the user device.

[0019] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0020] Memory and processor;

[0021] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described three-dimensional spatial map positioning method.

[0022] According to a fifth aspect of the embodiments of this specification, an augmented reality (AR) device or a virtual reality (VR) device is provided, comprising:

[0023] Memory, processor, and display;

[0024] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they perform the following steps:

[0025] Acquire multimedia data of the target environment and send the multimedia data to the server based on the extended reality framework;

[0026] Receive location information sent by the server, and determine the user's current location based on the location information;

[0027] Generate an AR environment image based on the user's current location; or

[0028] Based on the user's current location, corresponding reference information is obtained, wherein the reference information includes preset reference information and / or real-time reference information. The reference information is information for the user's reference. The preset reference information is reference information pre-entered in the user's device, and the real-time reference information is reference information displayed in real time according to the user's real-time location.

[0029] The AR environment image or the reference information is rendered onto the display of the augmented reality (AR) device or the virtual reality (VR) device for display.

[0030] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described three-dimensional spatial map positioning method.

[0031] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described three-dimensional spatial map positioning method.

[0032] The positioning system for a three-dimensional spatial map provided in this specification includes a data acquisition device, a user device, and a server. The data acquisition device is configured to acquire basic environmental data information of a target environment and send the basic environmental data information to the server, wherein the basic environmental data information includes multi-dimensional sensor data information. The server is configured to receive the basic environmental data information and generate a three-dimensional spatial map of the target environment based on the basic environmental data information. The user device is configured to acquire multimedia data of the target environment and send the multimedia data to the server based on an extended reality framework. The server is also configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user device.

[0033] The 3D spatial map positioning system provided in this manual involves a data acquisition device collecting multi-dimensional environmental data of the target environment. The server then constructs a 3D spatial map of the target environment based on this data, improving the accuracy of the 3D spatial map. The system utilizes an extended reality framework to facilitate interaction between the data acquisition device and the server, as well as between the user device and the server, achieving broad compatibility and resolving issues of low terminal compatibility and difficulty in adaptation. A visual positioning service deployed on the server enables positioning within the 3D spatial map. Coupled use of the visual positioning service with the extended reality framework further enhances the accuracy of 3D spatial positioning. Moreover, during the positioning process, there is no need to redeploy other conditions within the target environment, significantly reducing costs. Attached Figure Description

[0034] Figure 1 This is a structural block diagram of a three-dimensional spatial map positioning system provided in one embodiment of this specification;

[0035] Figure 2 This is a schematic diagram of the architecture of a visual positioning service technology provided in one embodiment of this specification;

[0036] Figure 3 This is a schematic diagram of an edge-cloud collaborative architecture that couples a visual positioning service with an extended reality framework, as provided in one embodiment of this specification.

[0037] Figure 4 This is a flowchart illustrating a three-dimensional spatial map positioning method provided in one embodiment of this specification;

[0038] Figure 5 This is a flowchart illustrating the processing steps of a three-dimensional spatial map localization method applied to a speed skating training scenario, as provided in one embodiment of this specification.

[0039] Figure 6This is a schematic diagram of the structure of a three-dimensional spatial map positioning device provided in one embodiment of this specification;

[0040] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0041] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0042] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0044] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0045] AR (Augmented Reality) navigation: AR navigation is a navigation mode that deeply integrates maps, mobile phone cameras or AR glasses cameras with AR technology / spatial mapping. The camera will display everything in the real world on the mobile phone screen, while virtual models such as cartoon characters and directional arrows will be superimposed on the existing image. This virtual model can guide pedestrians in the direction of navigation.

[0046] Employing 3D spatial reconstruction equipment and a cloud-based algorithm engine to achieve 3D spatial mapping, users can use smartphones with built-in IMUs or AR glasses cameras to identify their real-world environment and achieve real-time high-precision 3D spatial positioning for accurate visual navigation. Virtual navigation prompts are overlaid on the real-world environment to help users quickly find target parking spaces, racetracks, meeting rooms, spectator seats, shops, service desks, elevators, restrooms, attractions, etc., and can be integrated with navigation, interactive entertainment, and marketing applications. Application scenarios include: training venues, exhibitions, press conferences, shopping malls, scenic spots, industrial parks, exhibition halls, and museums.

[0047] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. It's a collective term for various technologies such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). By integrating the visual interaction technologies of these three technologies, it provides users with a seamless and immersive experience between the virtual and real worlds.

[0048] IMU (Inertial Measurement Unit): Used to measure an object's three-axis attitude angles (or angular rates) and acceleration. An IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers detect the object's acceleration signals along the three independent axes of the carrier's coordinate system, while the gyroscopes detect the carrier's angular velocity signals relative to the navigation coordinate system. By measuring the object's angular velocity and acceleration in three-dimensional space, the object's attitude can be calculated. It has significant applications in navigation.

[0049] GPS (Global Positioning System) is a high-precision radio navigation positioning system based on artificial Earth satellites. It can provide accurate geographical location, speed of movement, and precise time information anywhere in the world and in near-Earth space.

[0050] VPS (Visual Positioning Service): This refers to a system or service that uses image information for positioning. A VPS deployed in the cloud can provide large-scale 3D maps and map POI (Point of Information) markers, map semantic services, and global positioning services.

[0051] iBeacon: iBeacon technology refers to the use of Bluetooth Low Energy (BLE) technology. iBeacon base stations can automatically create a signal area. When a device enters this area, the corresponding application will prompt the user whether they want to connect to the network. iBeacon is a low-power Bluetooth technology that works similarly to previous Bluetooth technologies. iBeacon emits signals, and IOS (Internet Working Operating System - Cisco, abbreviated as IOS) devices locate, receive, and send back signals. Many corresponding indoor positioning applications can be developed based on this simple positioning technology.

[0052] Simultaneous Localization and Mapping (SLAM) is a technique for estimating sensor motion and reconstructing structures in unknown environments. Currently, the sensors used in SLAM are mainly divided into two categories: Lidar SLAM based on lidar and Visual SLAM based on vision.

[0053] Six Degrees of Freedom (6DOF): In addition to the ability to rotate along the X, Y, and Z axes, it also has the ability to move along the X, Y, and Z axes. 6DOF XR devices can simulate all head movements. Furthermore, using displacement data, the user's height can be determined, allowing users to visually perceive the actual height of objects by wearing 6DOF-enabled AR glasses, making the scene more realistic.

[0054] AR Tracking Technology: For AR glasses to achieve the fusion of the physical environment and virtual elements, they must be able to track the virtual environment. Visual tracking technologies include image-based tracking, object-based tracking, and simultaneous localization and mapping (SLAM). Image-based tracking first requires preprocessing the image to be tracked to obtain a set of feature points, and then matching the images in the video stream in real time. Object-based tracking is similar to image-based tracking; it first preprocesses the object to be tracked, storing its various features, and then analyzes the images in the video stream in real time to calculate the relative position between the tracking system and the tracked object. Simultaneous localization and mapping (SLAM) is a tracking technology that does not require preprocessing and can complete localization and mapping simultaneously after the start of the tracking process. While this method is simple to use, it cannot easily obtain the relative position between the glasses and a specific object in the physical environment as easily as the first two methods.

[0055] AR Interaction Technology: For AR glasses, the most convenient interaction methods are voice, gestures, and remote controls. Gesture interaction, because it doesn't require additional interaction devices—users only need to make gestures to input commands—has promoted the development of AR glasses interaction technology to some extent. Remote controls, as a traditional input method, are simple yet ensure accurate and efficient input; therefore, AR glasses products still come with remote controls or touchpads as interaction devices.

[0056] OpenXR: OpenXR is an XR application programming interface that provides high-performance access to augmented reality (AR) and virtual reality (VR) (collectively referred to as XR) platforms and devices.

[0057] Software Development Kit (SDK): In a broad sense, it refers to a collection of related documents, examples, and tools that assist in the development of a particular type of software. Generally, it is a collection of development tools used by software engineers to create application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.

[0058] This specification provides a positioning system for a three-dimensional spatial map, and also relates to a positioning method and apparatus for a three-dimensional spatial map, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments.

[0059] The following is an embodiment of a three-dimensional spatial map positioning system provided in this specification:

[0060] Figure 1 A structural block diagram of a positioning system for a three-dimensional spatial map provided according to an embodiment of this specification is shown.

[0061] The positioning system of the three-dimensional spatial map includes:

[0062] Data acquisition device 102, server 104, and user equipment 106;

[0063] The acquisition device 102 is configured to acquire basic environmental data information of the target environment and send the basic environmental data information to the server 104, wherein the basic environmental data information includes multi-dimensional sensor data information;

[0064] The server 104 is configured to receive the environmental basic data information and generate a three-dimensional spatial map of the target environment based on the environmental basic data information.

[0065] The user equipment 106 is configured to acquire multimedia data of the target environment and send the multimedia data to the server 104 based on the extended reality framework;

[0066] The server 104 is also configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user equipment 106.

[0067] The data acquisition device refers to the device used to collect basic environmental data information of the target environment. This data acquisition device includes a sensor collection platform, such as a panoramic camera. The target environment refers to the environment in which the user needs to be located. For example, if the user is currently in store 1 of shopping mall A and wants to go to store 2 of shopping mall A, but does not know how to get to store 2, then shopping mall A is the target environment.

[0068] Environmental basic data information refers to the basic environmental data information of the target environment collected by the acquisition equipment. Environmental basic data information is multi-dimensional sensing data information, which is data information in different dimensions, specifically including image information, distance information, location information, movement speed, etc. of the target environment.

[0069] Extended Reality (XR) is an open, cross-platform technology that combines real and virtual elements using computers, and is compatible with the vast majority of devices on the market. For example, an Extended Reality framework can be an XR platform, which runs on an XR chip, and the XR chip is installed in a smart device.

[0070] In practical applications, when the positioning system of a 3D spatial map needs to be updated, the updated positioning system of the 3D spatial map only needs to be sent to the XR platform. The XR platform runs on the XR chip, so that all devices with XR chips can update the positioning system of the 3D spatial map without having to adapt to different models of devices one by one.

[0071] The positioning system for the three-dimensional spatial map provided in this specification allows user equipment to upload collected data information to the server via an extended reality framework. This makes the positioning system for the three-dimensional spatial map unrestricted by device model. As long as the terminal is running an extended reality framework, the positioning system provided in the embodiments of this specification can be used, thereby enabling application on different terminals and achieving broad cross-terminal platform compatibility and support, solving the current problems of limited terminal compatibility and low adaptability.

[0072] The server is used to generate a 3D spatial map of the target environment based on the environmental baseline data and to locate the user's position based on visual positioning services. A 3D spatial map is a 3D spatial map of the target environment generated based on the environmental baseline data. Using the previous example, if the target environment is shopping mall A, then the 3D spatial map is a 3D spatial map of shopping mall A.

[0073] Specifically, after the data acquisition device collects basic environmental data of the target environment, it sends the basic environmental data to the server, and the server generates a three-dimensional spatial map of the target environment based on the basic environmental data.

[0074] User equipment refers to intelligent devices used to collect multimedia data of a target environment. This can be understood as smart cars, smartphones, smart glasses, smartwatches, or other smart wearable devices with image capture capabilities. Multimedia data can be understood as photos or videos of the target environment taken by the user through various intelligent devices with camera functions. This multimedia data is used to locate the user within the target environment. Continuing the example above, when the target environment is shopping mall A, the multimedia data would be photos or videos of shop number 1 in shopping mall A. Location information can be understood as the user's location within the target environment.

[0075] Specifically, in one embodiment provided in this specification, the 3D spatial mapping and 3D spatial positioning services in the server are encapsulated into a visual positioning service. The user device collects multimedia data of the target environment and sends the multimedia data to the server based on the extended reality framework. The server can determine the user's location information based on the generated 3D spatial map, multimedia data, and visual positioning service, and send the location information to the user device so that the user knows their location information.

[0076] In practical applications, users often face situations where they don't know where their destination is or how to get there. In such cases, users typically use navigation to solve this problem. However, existing 2D map navigation methods can cause some users with weaker sense of direction to have difficulty understanding and using maps effectively. Based on this, 3D spatial map navigation has gradually emerged. However, the accuracy of existing 3D spatial map navigation is not high, making it difficult for users to find their destination as desired.

[0077] Therefore, this specification provides a positioning system for a three-dimensional spatial map, which collects multi-dimensional environmental basic data information of the target environment through a data acquisition device and uploads the environmental basic data information to a server. After receiving the environmental basic data information, the server generates a three-dimensional spatial map of the target environment based on the environmental basic data information.

[0078] Compared to traditional 2D maps, 3D spatial maps can more intuitively display the target environment to users, solving the problem of users having difficulty understanding maps. Generating 3D spatial maps of the target environment based on multi-dimensional environmental basic data information can improve the accuracy of the generated 3D spatial maps, reduce errors, and improve the user experience.

[0079] The aforementioned 3D spatial map positioning system can also acquire multimedia data of the target environment through user equipment and upload the multimedia data to the server based on the extended reality framework. The server determines the user's location information based on the generated 3D spatial map, the multimedia data of the target environment, and the visual positioning service, and sends the user's location information to the user equipment.

[0080] In practical applications, users can obtain multimedia data of the target environment through their user devices. The user devices then upload the multimedia data to the server based on the extended reality framework. The server determines the user's location information based on the 3D spatial map, the multimedia data of the target environment, and visual positioning services, and sends the location information to the user devices. At this point, the user can determine their location in the target environment based on the location information.

[0081] For example, if user Y is located in store number 1 of shopping mall A, user Y can take a photo of store number 1 with their mobile phone. The mobile phone will then upload the photo of store number 1 to the server via an augmented reality frame. After receiving the photo of store number 1, the server can determine user Y's location information in shopping mall A based on the AR map of shopping mall A, the photo of store number 1, and visual positioning services. The server will then send user Y's location information in shopping mall A to the mobile phone and display the AR map containing user Y's location information in store number 1 on user Y's mobile phone.

[0082] Combined with appendix Figure 2 Let's take a look, attached Figure 2 This is a schematic diagram of the architecture of a visual positioning service technology provided in this specification. After receiving the basic environmental data of the target environment, the server preprocesses the basic environmental data and constructs a three-dimensional spatial map based on large-scene mapping algorithms and services. It can provide multiple positioning services based on vision, semantics, Bluetooth, and GPS, which greatly improves the accuracy of the generated three-dimensional spatial map. Based on the generated three-dimensional spatial map, positioning algorithms, such as 3D visual positioning algorithms, are used to calculate the user's position in real time, thereby determining the user's location information. The user's location information is then sent to the user device in the form of a software development kit (SDK).

[0083] Taking a positioning algorithm as an example, the process of determining a user's location information is explained. The server preprocesses the generated 3D spatial map, which can be divided into several smaller maps. Each smaller map includes a panoramic view of the target environment and views from different angles. Based on the multimedia data of the target environment, image retrieval is performed on these smaller maps, and a rough location is achieved based on the similarity between the multimedia data and each smaller map. Local image features are extracted from each frame of multimedia data image, and feature matching is performed between the local features of each frame of multimedia data image and the smaller maps with high similarity, thus completing precise positioning and determining the user's location information.

[0084] The positioning system for a three-dimensional spatial map provided in this specification includes a data acquisition device, a user device, and a server. The data acquisition device is configured to acquire basic environmental data information of a target environment and send the basic environmental data information to the server, wherein the basic environmental data information includes multi-dimensional sensor data information. The server is configured to receive the basic environmental data information and generate a three-dimensional spatial map of the target environment based on the basic environmental data information. The user device is configured to acquire multimedia data of the target environment and send the multimedia data to the server based on an extended reality framework. The server is also configured to determine the user's location information based on the three-dimensional spatial map, the multimedia data, and a visual positioning service, and send the location information to the user device.

[0085] The 3D spatial map positioning system provided in this manual involves a data acquisition device collecting multi-dimensional environmental data of the target environment. The server then constructs a 3D spatial map of the target environment based on this data, improving the accuracy of the 3D spatial map. The system utilizes an extended reality framework to facilitate interaction between the data acquisition device and the server, as well as between the user device and the server, achieving broad compatibility and resolving issues of low terminal compatibility and difficulty in adaptation. A visual positioning service deployed on the server enables positioning within the 3D spatial map. Coupled use of the visual positioning service with the extended reality framework further enhances the accuracy of 3D spatial positioning. Moreover, during the positioning process, there is no need to redeploy other conditions within the target environment, significantly reducing costs.

[0086] To improve the accuracy of the three-dimensional spatial map, in one optional embodiment provided in this specification, the acquisition device 102 is further configured to acquire the basic environmental data information through a sensor platform running in the acquisition device 102, wherein the sensor platform includes at least two of a visual sensor, a laser sensor, an inertial measurement unit, and a global positioning system.

[0087] The data acquisition equipment includes a sensor aggregation platform, which can be understood as a collection of multiple sensors. This platform includes at least two of the following: a vision sensor, a laser sensor, an inertial measurement unit (IMU), and a global positioning system (GPS). The vision sensor is used to acquire image information of the target environment; the laser sensor is used to acquire distance information of the target environment; the IMU is used to measure the three-axis attitude angles and accelerations of buildings or objects in the target environment; and the GPS is used to provide accurate geographic location, speed, and time information.

[0088] The data acquisition device provided in this manual collects basic environmental data of the target environment through the aforementioned sensor platform. This basic environmental data includes information in various dimensions such as image, distance, acceleration, velocity, and time. By providing basic environmental data in different dimensions, a three-dimensional spatial map can be generated, thereby improving the accuracy of the three-dimensional spatial map.

[0089] Not all collected environmental baseline data can be used to generate a 3D spatial map of the target environment. This data inevitably contains duplicate or erroneous information. Directly generating a 3D spatial map from this data would require processing duplicates and errors, which would not only prolong the generation time but also reduce the accuracy of the resulting 3D spatial map. Therefore, to ensure the accuracy and improve the efficiency of 3D spatial map generation, preprocessing of the collected environmental baseline data is necessary before generating the corresponding 3D spatial map.

[0090] Based on this, in one optional embodiment provided in this specification, the server 104 is further configured to perform data preprocessing on the environmental basic data information to obtain an initial three-dimensional spatial map of the target environment, and to determine the three-dimensional spatial map based on the initial three-dimensional spatial map, wherein the accuracy of the three-dimensional spatial map is higher than that of the initial three-dimensional spatial map.

[0091] The initial 3D spatial map is a 3D spatial map obtained after preprocessing the basic environmental data, and its accuracy is relatively low. Data preprocessing refers to processing the basic environmental data, such as data registration, coordinate correction, and data simplification.

[0092] After receiving the basic environmental data, the server performs preprocessing such as data registration, coordinate correction, and data simplification, and generates an initial 3D spatial map of the target environment based on the preprocessed data. After obtaining the initial 3D spatial map of the target environment, a final 3D spatial map is determined based on the initial 3D spatial map.

[0093] Specifically, the process of obtaining a 3D spatial map from an initial 3D spatial map is as follows:

[0094] In one optional embodiment provided in this specification, the server 104 is further configured to eliminate the accumulated error in the initial three-dimensional spatial map based on a loop closure detection method to obtain the three-dimensional spatial map.

[0095] Among them, the loop closure detection method refers to the method of detecting whether the map can be successfully closed, thereby reducing the cumulative error in the initial 3D spatial map. Loop closure detection makes the detected map more accurate.

[0096] Specifically, loop closure detection can be implemented in the following way:

[0097] The target keyframes of the initial 3D spatial map are determined, the bag-of-words vectors of the target keyframes are determined based on the bag-of-words model, the candidate image sequence of the target keyframes is determined, the similarity between the target keyframes and each candidate image frame is calculated, and loop closure pairs are obtained according to the calculation results. If the loop closure pairs are successfully verified, the 3D spatial map is determined.

[0098] The target keyframe refers to the keyframe selected from a queue of keyframes for loop closure detection. The bag-of-words model, a simplified representation model used in natural language processing and information retrieval, can convert the feature point descriptors of a single image frame into bag-of-words descriptive vectors for direct comparison with another image frame, or compare an image frame with a bag-of-words database, returning the top few most similar images. A loop closure pair consists of the target keyframe and a candidate image frame with the highest similarity to the target keyframe.

[0099] Specifically, the target keyframe is input into the bag-of-words model to obtain the bag-of-words vector of the target keyframe. Based on the bag-of-words vector, the candidate image frame sequence of the target keyframe is determined. The similarity between the target keyframe and each candidate image frame is calculated. The candidate image frame with the highest similarity is paired with the target keyframe to form a loop closure pair, and loop closure verification is performed. If the loop closure pair verification is successful, the 3D spatial map is determined.

[0100] By performing loop closure detection on the initial 3D spatial map, the accumulated errors of the initial 3D spatial map can be eliminated, thereby improving the accuracy of the initial 3D spatial map.

[0101] In practical applications, we use 3D spatial maps with location information annotations, and we also need 3D spatial maps without location information annotations. For example, when a user needs to accurately find a certain location, a 3D spatial map with location information annotations, such as seats, floors, and doors, is required; when a user is in a navigation scenario and only needs to know the direction of travel, a 3D spatial map without location information annotations can meet the user's needs.

[0102] Therefore, in one optional embodiment provided in this specification, the server 104 is further configured to receive the location annotation data of the target environment and annotate the three-dimensional spatial map based on the location annotation data.

[0103] Location annotation data refers to location data information used to annotate 3D spatial maps. This location annotation data can be manually entered by relevant technical personnel or obtained from 2D maps. This manual does not impose any restrictions on the type or acquisition method of location annotation data. The following explanation uses two methods—directly usable location annotation data and data obtained from 2D maps—as examples.

[0104] If the location annotation data is readily available, it can be directly annotated on the 3D spatial map. Location annotation data can be obtained in the following ways:

[0105] In one optional embodiment provided in this specification, the server is further configured to receive an input instruction for location annotation data of the target environment, open an input interface in response to the input instruction, and receive location annotation data of the target environment based on the input interface.

[0106] Among them, the input instruction for the location annotation data of the target environment refers to the instruction used to input the location annotation data corresponding to the target environment, and the input interface refers to the input interface used to input the location annotation data of the target environment. For example, if you need to mark the location of Shop No. 1 on the generated 3D spatial map, you can input "Shop No. 1" at the location corresponding to Shop No. 1. The display interface used to input "Shop No. 1" is the input interface for "Shop No. 1".

[0107] Specifically, in order to annotate the location data of each location in the target environment, you can trigger the input command for the location annotation data of the target environment, and enter the location annotation data of the corresponding location in the target environment in the opened input interface. After the input is completed, a three-dimensional spatial map with location annotation data can be generated.

[0108] If the location annotation data is obtained from a planar map, in one optional embodiment provided in this specification, the system further includes a planar map acquisition device;

[0109] The planar map acquisition device is configured to acquire a planar map corresponding to the target environment and send the planar map to the server 104, wherein the planar map carries location data;

[0110] The server 104 is also configured to receive the planar map, obtain location data in the planar map, and annotate the three-dimensional spatial map based on the location data of the planar map.

[0111] Among them, the planar map acquisition device refers to the device used to acquire a planar map corresponding to the target environment. The planar map acquisition device can be the acquisition device mentioned above, or other devices with acquisition functions besides the acquisition device mentioned above. This specification does not limit it here.

[0112] Specifically, the planar map acquisition device acquires a planar map corresponding to the target environment and sends the acquired planar map to the server. The server receives the planar map corresponding to the target environment, obtains the location data carried in the planar map, and marks the obtained location data on the three-dimensional spatial map.

[0113] The 3D spatial map positioning system provided in this manual can generate both 3D spatial maps without location information annotations and those with location information annotations, thus meeting different user needs. It also offers different methods for acquiring location annotation data, making the annotation of 3D spatial maps quite flexible.

[0114] In one optional embodiment provided in this specification, the user equipment 106 is further configured to invoke an image acquisition unit to acquire image data of the target environment, wherein the image acquisition unit is configured in the user equipment.

[0115] The image acquisition unit is a device with image acquisition capabilities, such as a camera. Image data refers to data displayed in image form, such as photos taken by a camera or video frames included in a video recording. Specifically, the user device acquires image data of the target environment by calling the image acquisition unit. Taking a smartphone as an example, the user can use the camera on their phone to take pictures or videos of the target environment.

[0116] Based on the image data of the target environment collected by the user equipment, the server can further determine the user's location information on the basis of the 3D spatial map and visual positioning service, thereby accurately realizing the positioning function of the 3D spatial map positioning system.

[0117] Furthermore, the positioning system of the three-dimensional spatial map provided in this specification can be applied to navigation scenarios. Therefore, in an optional embodiment provided in this specification, the user equipment 106 is further configured to determine the user's current location based on the location information, receive an input instruction, determine a target location in response to the input instruction, determine a navigation path based on the target location and the current location, and generate and display an AR navigation image based on the navigation path.

[0118] The target location refers to the final location in a navigation cycle, which is the location entered by the user on their device. The input command is a command generated by the user by triggering the target location input control, or a command obtained from voice information. The navigation path is the navigation path from the current location to the target location. For example, if the current location is P and the target location is P1, then the corresponding navigation path is the navigation path from P to P1.

[0119] Specifically, the user device determines the user's current location based on location information, determines the target location in response to the input command for the target location, generates a navigation path from the current location to the target location based on the current location and the target location, and displays it on the user device in the form of an AR navigation image.

[0120] Taking a smartphone as an example, the smartphone receives location information sent by the server and determines the user's current location as P. In response to the input instruction of the target location P1, it determines the target location as P1. Based on the current location P and the target location P1, it generates a navigation path from the current location P to the target location P1 and displays it on the smartphone in the form of an AR navigation image.

[0121] In real-world AR navigation scenarios, there are situations where users put down their smart devices or turn off the camera. In such cases, the ongoing navigation path is forcibly interrupted, and the user needs to manually refresh and reposition to resume navigation. This repositioning process can take several seconds or even minutes, causing inconvenience for users and degrading their user experience.

[0122] The 3D spatial map positioning system provided in this manual uses iBeacon technology to set points on a planar map of the target environment and assign a point marker to each point. The user device obtains its position information on the planar map through the point markers, and then determines its position information on the 3D spatial map based on the position information on the planar map, completing the repositioning and displaying it again on the user device in the form of an AR navigation image. The 3D spatial map positioning system provided in this manual, through the combination of iBeacon technology and AR navigation technology, improves the efficiency of determining the user's location information and enhances the user experience.

[0123] The 3D spatial map positioning system provided in this manual can provide users with realistic AR navigation images in real time when applied to navigation scenarios. Furthermore, the 3D spatial map generated by the positioning system provided in this manual has high accuracy, thus improving the user experience.

[0124] To facilitate user access and reduce network latency, the positioning system for the 3D spatial map may include an edge server. After the server generates the 3D spatial map, it is downloaded to the edge server. In practical applications, the corresponding edge server can be determined based on the network location information of the user's device, and the edge server can then locate the user's position.

[0125] In one optional embodiment provided in this specification, the server 104 includes a cloud server and multiple edge servers;

[0126] The cloud server is configured to obtain the network location information of the user device 106, determine the target edge server from the plurality of edge servers based on the network location information, and send the three-dimensional spatial map to the target edge server.

[0127] The target edge server is configured to receive and store the three-dimensional spatial map;

[0128] The user equipment 106 is also configured to acquire multimedia data of the target environment and send the multimedia data to the target edge server based on the extended reality framework;

[0129] The target edge server is also configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user equipment 106.

[0130] Among them, edge servers are computers that exist at the logical "edge" of the network; network location information can be understood as information used to determine the network location of user devices, which can be IP address, MAC address, etc.

[0131] In practical applications, several edge servers are distributed in the network environment. In order to reduce network latency and improve response rate, it is necessary to select the edge server closest to the user device as the target edge server.

[0132] Specifically, the cloud server obtains the network location information of the user device, identifies the edge server closest to the user device's network location as the target edge server (where network location refers to the logical network location), and sends the generated 3D spatial map to the target edge server, which then receives and stores the 3D spatial map.

[0133] At this point, after acquiring the multimedia data of the target environment, the user device will no longer send it to the server, but to the target edge server. The target edge server determines the user's location information based on the stored 3D spatial map, the multimedia data of the target environment, and the visual positioning service, and then sends the user's location information to the user device.

[0134] For example, after obtaining the network location information of user device S, the cloud server determines the target edge server as M based on the network location information and sends a 3D spatial map to the target edge server M. The target edge server receives the multimedia data of the target environment uploaded by user device S, determines the user's location information based on the multimedia data, the 3D spatial map and the visual positioning service, and sends the location information to user device S so that the user knows his or her location in the target environment.

[0135] In the embodiments described above in this specification, by introducing an edge server, the 3D spatial map can be accessed from the edge server after it is generated, which can reduce network latency compared to a server.

[0136] After achieving three-dimensional spatial positioning, the acquired location information also has its value and role, and different application scenarios can realize different application values.

[0137] In one optional embodiment provided in this specification, the user equipment 106 is further configured to receive the location information, determine the user's current location based on the location information, and obtain and display corresponding reference information according to the user's current location. The reference information includes preset reference information and / or real-time reference information. The reference information is information for user reference. The preset reference information is reference information pre-entered into the user equipment, and the real-time reference information is reference information displayed in real time according to the user's real-time location.

[0138] Reference information refers to information used by users for reference and to make corresponding changes based on the reference information. This reference information can be preset reference information or real-time reference information. Accordingly, preset reference information is reference information that has been pre-entered into the user's device, while real-time reference information is reference information that is displayed in real time based on the user's real-time location.

[0139] For example, in speed skating training, the virtual coach's posture and movement information recorded in the AR glasses worn by the athlete can serve as preset reference information. This preset reference information is displayed in real time based on the athlete's skating position, prompting the athlete to correct incorrect skating methods or movements in a timely manner. Meanwhile, the athlete's posture and movement information displayed in real time in the AR glasses as the athlete skates serves as real-time reference information.

[0140] For example, in automotive applications, there is pre-recorded information such as different driving speeds for different road sections. This information is called preset reference information, which is used to remind the driver to slow down, etc. The actual real-time speed of the car is the real-time reference information.

[0141] Specifically, the user equipment determines the user's current location based on location information and obtains and displays corresponding reference information in real time based on the user's current location.

[0142] Taking speed skating training as an example, the AR glasses worn by the athlete during training determine that the athlete is skating on a curve based on the location information, and obtain and display the training posture of the virtual coach when skating on the curve. At this time, the athlete can make corresponding adjustments based on the training posture of the virtual coach.

[0143] Alternatively, the training postures of the virtual coach and the real-time gliding postures of the athlete can be displayed simultaneously on AR glasses.

[0144] Taking a car driving scenario as an example, the dashboard obtains and displays the standard driving speed and the actual real-time speed of the car based on its current location. The driver can then make appropriate adjustments to the actual real-time speed based on the standard driving speed and the actual real-time speed.

[0145] The positioning system of the three-dimensional spatial map provided in this manual is not limited to positioning or navigation; it can also prompt users to make corresponding adjustments to the current operating status to improve the user experience.

[0146] In practical applications, such as fitness and speed skating, there is also a need to analyze and summarize real-time data in order to make better usage plans for future use.

[0147] In one optional embodiment provided in this specification, the positioning system of the three-dimensional spatial map further includes a data analysis device;

[0148] The user equipment 106 is further configured to acquire the user's data information to be analyzed and send the data information to be analyzed to the data analysis device;

[0149] The data analysis device is configured to perform data analysis on the data to be analyzed and obtain data analysis results.

[0150] The data analysis equipment can be intelligent devices with data analysis capabilities, such as smartphones, tablets, and laptops. The data to be analyzed refers to the data information that needs to be analyzed, such as athletes' training data or fitness enthusiasts' fitness data.

[0151] For example, when user Zhang San runs outdoors, the real-time running data generated during his run can be used as data for analysis, as can his historical running data. Specifically, taking an outdoor running scenario as an example, the real-time running data generated during Zhang San's run is collected by his smart bracelet and sent to his mobile phone. After receiving the data, the phone analyzes it and outputs the results. After finishing his run, Zhang San can view the data and analysis results on his phone. Users can use these data analysis results as a reference for developing future running plans, enabling them to run more healthily and rationally.

[0152] Furthermore, taking speed skating as an example, this specification provides an optional implementation method.

[0153] The user equipment 106 is further configured to acquire the user's gliding data information and send the gliding data information to the data analysis device, wherein the gliding data information is training data generated during gliding training;

[0154] The data analysis device is configured to perform data analysis on the gliding data information, and obtain and display the user's gliding data analysis results.

[0155] Among them, skating data information refers to the skating training data information generated by users during skating on the speed skating rink, while skating data analysis results refer to the data analysis results generated after analyzing the skating data information generated by users.

[0156] Based on the same principle as above, after acquiring the user's gliding data information, the user equipment sends the user's gliding data information to the data analysis device. The data analysis device performs data analysis on the user's gliding data information and generates the corresponding data analysis results.

[0157] The positioning system for the three-dimensional spatial map provided in this manual can perform data analysis on the user's data to be analyzed and output the data analysis results for subsequent use, providing convenience for the user.

[0158] The positioning system for a three-dimensional spatial map provided in this specification includes a data acquisition device, a user device, and a server. The data acquisition device is configured to acquire basic environmental data information of a target environment and send the basic environmental data information to the server, wherein the basic environmental data information includes multi-dimensional sensor data information. The server is configured to receive the basic environmental data information and generate a three-dimensional spatial map of the target environment based on the basic environmental data information. The user device is configured to acquire multimedia data of the target environment and send the multimedia data to the server based on an extended reality framework. The server is also configured to determine the user's location information based on the three-dimensional spatial map, the multimedia data, and a visual positioning service, and send the location information to the user device.

[0159] The 3D spatial map positioning system provided in this manual involves a data acquisition device collecting multi-dimensional environmental data of the target environment. The server then constructs a 3D spatial map of the target environment based on this data, improving the accuracy of the 3D spatial map. The system utilizes an extended reality framework to facilitate interaction between the data acquisition device and the server, as well as between the user device and the server, achieving broad compatibility and resolving issues of low terminal compatibility and difficulty in adaptation. A visual positioning service deployed on the server enables positioning within the 3D spatial map. Coupled use of the visual positioning service with the extended reality framework further enhances the accuracy of 3D spatial positioning. Moreover, during the positioning process, there is no need to redeploy other conditions within the target environment, significantly reducing costs.

[0160] Edge servers can also be deployed in the 3D spatial map positioning system provided in this manual. Deploying edge servers can reduce network latency and improve response speed. Alternatively, data analysis devices can be deployed in the 3D spatial map positioning system to analyze the data information that needs to be analyzed during the positioning process and output the data analysis results to the user, thereby improving the user experience.

[0161] The following is in conjunction with the appendix Figure 3Taking the application of the three-dimensional spatial map positioning system provided in this specification in speed skating training as an example, the positioning system of the three-dimensional spatial map will be further explained. Figure 3 This specification illustrates a schematic diagram of an edge-cloud collaborative architecture that couples a visual positioning service with an extended reality framework, as provided in one embodiment of this specification.

[0162] The positioning system of the three-dimensional spatial map includes:

[0163] Panoramic camera 302, server 304, and AR glasses 306;

[0164] The panoramic camera 302 is configured to collect basic environmental data information of the speed skating training venue through a sensor collection platform running in the panoramic camera 302, and send the basic environmental data information to the server 304, wherein the basic environmental data information includes multi-dimensional sensor data information.

[0165] The server 304 is configured to receive the environmental basic data information, perform data preprocessing on the environmental basic data information to obtain an initial three-dimensional spatial map of the speed skating training venue, determine a three-dimensional spatial map of the speed skating training venue based on the initial three-dimensional spatial map, receive the location annotation data of the speed skating training venue, and annotate the three-dimensional spatial map based on the location annotation data.

[0166] The AR glasses 306 are configured to call the AR tracking camera to collect image data of the speed skating training venue and send the image data to the server 304 based on the extended reality framework;

[0167] The server 304 is also configured to determine the athlete's location information based on the three-dimensional spatial map, the image data, and the visual positioning service, and send the location information to the AR glasses 306;

[0168] The AR glasses 306 are also configured to receive the location information, determine the athlete's current location based on the location information, and obtain and display corresponding reference information based on the athlete's current location, wherein the reference information includes preset reference information and / or real-time reference information.

[0169] Specifically, the panoramic camera collects basic environmental data of the speed skating training venue through a sensor platform running within the panoramic camera, and sends the basic environmental data to the server; the server preprocesses the basic environmental data and obtains an initial three-dimensional spatial map of the speed skating training venue based on the preprocessed data, and determines the three-dimensional spatial map based on the initial three-dimensional spatial map.

[0170] Athletes can wear AR glasses during training skating. The AR glasses collect image data of the speed skating training venue by calling AR tracking cameras, and upload the image data to the server through an extended reality framework. The server determines the athlete's position information based on the generated 3D spatial map, image data and visual positioning service deployed on the server. For example, if the athlete is skating on a curve, the athlete's position information is sent to the AR glasses worn by the athlete. Based on the athlete's position information, the corresponding reference information is obtained and displayed on the lenses of the AR glasses.

[0171] For example, if an athlete is gliding on a curve, the AR glasses will display the athlete's real-time gliding information, which may include the athlete's gliding posture, gliding speed, heart rate, etc. At the same time, the AR glasses can also display the gliding posture of a virtual coach on the curve.

[0172] Correspondingly, the information about the athlete and the virtual coach displayed on the AR glasses lenses allows the athlete to understand their skating information in real time and make corresponding adjustments based on the virtual coach's skating posture. Training with AR glasses can complete athlete training with reduced manpower and material resources.

[0173] Furthermore, in order to reduce the impact of network latency, facilitate coaches in analyzing athletes' training data, and develop subsequent training plans for athletes, edge servers and data analysis devices can be deployed in the positioning system of the three-dimensional spatial map. In one optional embodiment provided in this specification, the server 304 includes a cloud server and multiple edge servers, and the positioning system of the three-dimensional spatial map also includes data analysis devices.

[0174] The cloud server is also configured to obtain the network location information of the AR glasses 306, determine the target edge server from the plurality of edge servers based on the network location information, and send the three-dimensional spatial map to the target edge server;

[0175] The target edge server is configured to receive and store the three-dimensional spatial map;

[0176] The AR glasses 306 are also configured to acquire image data of the speed skating training venue and send the image data to the target edge server based on the extended reality framework;

[0177] The target edge server is also configured to determine the athlete's location information based on the three-dimensional spatial map, the image data, and the visual positioning service, and send the location information to the AR glasses 306;

[0178] The AR glasses 306 are also configured to receive the location information, determine the athlete's current location based on the location information, and obtain and display corresponding reference information based on the athlete's current location, wherein the reference information includes preset reference information and / or real-time reference information;

[0179] The AR glasses 306 are also configured to acquire the athlete's data information to be analyzed and send the data information to be analyzed to the data analysis device;

[0180] The data analysis device is configured to perform data analysis on the data to be analyzed and obtain data analysis results.

[0181] Specifically, the cloud server determines the target edge server based on the AR glasses' network location information and sends the generated 3D spatial map of the speed skating training venue to the target edge server. At this point, after acquiring the image data of the speed skating training venue, the AR glasses send the image data to the target edge server via an extended reality framework. The target edge server determines the athlete's skating position information based on the 3D spatial map, image data, and visual positioning service. Upon receiving the skating position information, the AR glasses determine the athlete's skating position within the training venue, obtain the corresponding real-time skating information of the athlete and the skating posture of the virtual coach, and display them on the lenses of the AR glasses so that the athlete can adjust their skating posture in a timely manner based on the virtual coach's skating posture.

[0182] During the above process, AR glasses can also send the training data that the athlete needs to analyze to the data analysis device. The data analysis device analyzes the training data and obtains the data analysis results. These results can be used by coaches to provide corresponding training guidance to athletes or to develop more suitable training plans for athletes in subsequent training.

[0183] In one optional implementation provided in the embodiments of this specification, athletes can also communicate with coaches, teammates, etc. through the voice interaction function of AR glasses.

[0184] The three-dimensional spatial map positioning system provided in this manual can be applied to speed skating training scenarios. Using the three-dimensional spatial map positioning system for speed skating training can replace the role of the trainer currently required in training, reducing manpower and material resources, while also improving the efficiency of collecting athlete training data and enhancing training effectiveness.

[0185] See Figure 4 , Figure 4A flowchart of a three-dimensional spatial map positioning method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0186] Step 402: Receive the environmental basic data information uploaded by the acquisition device, and generate a three-dimensional spatial map of the target environment based on the environmental basic data information.

[0187] In one optional embodiment provided in this specification, the three-dimensional spatial map positioning method is applied to a server. Specifically, the server receives basic environmental data information of the target environment uploaded by the acquisition device, and generates a three-dimensional spatial map of the target environment based on the basic environmental data information.

[0188] In practical applications, not all collected environmental baseline data can be used to generate a 3D spatial map of the target environment. This data inevitably contains duplicate or erroneous information. Directly generating a 3D spatial map from this data would require processing duplicates and errors, which would not only prolong the generation time but also reduce the accuracy of the resulting 3D spatial map. Therefore, to ensure the accuracy of the 3D spatial map and improve its generation efficiency, preprocessing of the collected environmental baseline data is necessary before generating the corresponding 3D spatial map.

[0189] Based on this, in one optional implementation provided by the embodiments of this specification, generating a three-dimensional spatial map of the target environment based on the environmental basic data information includes:

[0190] The environmental basic data information is preprocessed to obtain an initial three-dimensional spatial map of the target environment;

[0191] The three-dimensional spatial map is determined based on the initial three-dimensional spatial map, wherein the accuracy of the three-dimensional spatial map is higher than that of the initial three-dimensional spatial map.

[0192] Specifically, after receiving the basic environmental data, the server performs preprocessing such as data registration, coordinate correction, and data simplification, and generates an initial 3D spatial map of the target environment based on the preprocessed data. After obtaining the initial 3D spatial map of the target environment, a final 3D spatial map is determined based on the initial 3D spatial map.

[0193] In one optional implementation provided in this specification, determining the three-dimensional spatial map based on the initial three-dimensional spatial map includes:

[0194] The accumulated errors in the initial 3D spatial map are eliminated using a loop closure detection method to obtain the 3D spatial map.

[0195] The loop closure detection method refers to the method of detecting whether a map can successfully close loops, thereby reducing the accumulated errors in the initial 3D spatial map. Loop closure detection improves the accuracy of the detected map. The loop closure detection method has been described in detail in the aforementioned 3D spatial map positioning system and will not be repeated here.

[0196] Step 404: Receive multimedia data of the target environment uploaded by the user equipment based on the extended reality framework.

[0197] Step 406: Determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user device.

[0198] Specifically, in one embodiment provided in this specification, the three-dimensional spatial mapping and three-dimensional spatial positioning services in the server are encapsulated into a visual positioning service. The server can determine the user's location information based on the generated three-dimensional spatial map, multimedia data, and visual positioning service, and send the location information to the user's device so that the user knows their location information.

[0199] In practical applications, we use 3D spatial maps with location information annotations, and we also need 3D spatial maps without location information annotations. For example, when a user needs to accurately find a certain location, a 3D spatial map with location information annotations, such as seats, floors, and doors, is required; when a user is in a navigation scenario and only needs to know the direction of travel, a 3D spatial map without location information annotations can meet the user's needs.

[0200] Therefore, in one optional embodiment provided in this specification, the method further includes:

[0201] Receive the location annotation data of the target environment;

[0202] The three-dimensional spatial map is labeled based on the location labeling data.

[0203] Specifically, after generating a 3D spatial map of the target environment, the server receives the location annotation data of the target environment and annotates the 3D spatial map based on the location annotation data.

[0204] Labeling a 3D spatial map based on each location is relatively labor-intensive. If the target environment has a planar map, the labeling can be done directly based on the planar map, which can improve the labeling efficiency of the 3D spatial map.

[0205] In one optional implementation provided in this specification, receiving the location annotation data of the target environment includes:

[0206] Receive a planar map of the target environment;

[0207] Accordingly, annotating the three-dimensional spatial map based on the location annotation data includes:

[0208] Obtain location data from the planar map, and annotate the three-dimensional spatial map based on the location data from the planar map.

[0209] Specifically, the location data of the target environment is extracted from the planar map, and the three-dimensional spatial map is annotated based on the location data of the planar map.

[0210] The 3D spatial map positioning system provided in this manual can generate both 3D spatial maps without location information annotations and those with location information annotations, thus meeting different user needs. It also offers different methods for acquiring location annotation data, making the annotation of 3D spatial maps quite flexible.

[0211] To facilitate user access and reduce network latency, the server can also send the generated 3D spatial map to the corresponding edge server. In practical applications, the edge server can then perform the subsequent use of the 3D spatial map.

[0212] In one optional implementation provided in the embodiments of this specification, the method further includes:

[0213] Obtain the network location information of the user equipment;

[0214] The target edge server is determined based on the network location information, and the three-dimensional spatial map is sent to the target edge server.

[0215] Specifically, the server determines the target edge server based on the network location information of the user device and sends the generated 3D spatial map to the target edge server, which can then store the 3D spatial map for later use.

[0216] The 3D spatial map positioning method provided in this manual improves the accuracy of the 3D spatial map by receiving multi-dimensional environmental basic data information uploaded by the acquisition device and constructing a 3D spatial map of the target environment using environmental basic data information of different dimensions. It achieves broad compatibility of the 3D spatial map positioning system by realizing the interaction process between the user device and the server through an extended reality framework, solving the problems of low terminal compatibility and difficult adaptation. The visual positioning service deployed on the server enables positioning within the 3D spatial map. Coupled use of the visual positioning service and the extended reality framework improves the accuracy of 3D spatial positioning, and during the positioning process, there is no need to deploy other conditions in the target environment again, greatly reducing costs. The server can also determine the target edge server based on the network location information of the user device. After generating the 3D spatial map, the target edge server interacts with the user device, thereby reducing network latency and improving response speed.

[0217] The following is in conjunction with the appendix Figure 5 Taking the application of the three-dimensional spatial map positioning method provided in this specification in a speed skating scenario as an example, the positioning method of the three-dimensional spatial map will be further explained. Among them, Figure 5 This specification illustrates a flowchart of a three-dimensional spatial map localization method for a speed skating training scenario, provided by an embodiment of this specification. The three-dimensional spatial map localization method is applied to a server and specifically includes the following steps.

[0218] Step 502: Receive the basic environmental data information of the speed skating training venue uploaded by the three-dimensional spatial reconstruction acquisition device.

[0219] Specifically, it receives image information, distance information, acceleration information, movement speed and time information of the speed skating training venue uploaded by panoramic cameras.

[0220] Step 504: Perform data preprocessing on the environmental basic data information to obtain an initial three-dimensional spatial map of the speed skating training venue.

[0221] Specifically, data registration, coordinate correction, and data simplification are performed on image information, distance information, acceleration information, movement speed and time information, and an initial three-dimensional spatial map of the speed skating training venue is obtained based on the processed data.

[0222] Step 506: Eliminate the accumulated errors in the initial 3D spatial map based on the loop closure detection method to obtain a 3D spatial map.

[0223] Specifically, the loop closure method is used to eliminate the accumulated errors in the initial three-dimensional spatial map to obtain a three-dimensional spatial map of the speed skating training venue.

[0224] Step 508: Receive multimedia data of the speed skating training venue uploaded by the AR glasses based on the extended reality framework.

[0225] During training, athletes can wear AR glasses to train, specifically receiving image or video data of the speed skating training venue uploaded by the AR glasses based on an extended reality framework.

[0226] Step 510: Determine the athlete's location information based on the three-dimensional spatial map, the multimedia data, and the visual positioning service, and send the location information to the AR glasses.

[0227] Specifically, the gliding position information of athletes is determined based on three-dimensional spatial maps, image data or video data and visual positioning services, and the gliding position information is sent to AR glasses.

[0228] When the three-dimensional spatial map positioning method provided in this manual is applied to speed skating training scenarios, it improves the accuracy of the three-dimensional spatial map by receiving basic environmental data of the speed skating training venue and generating a three-dimensional spatial map of the speed skating training venue. It also receives multimedia data of the speed skating training venue uploaded by AR glasses, and determines the skating position of the athlete through the three-dimensional spatial map, multimedia data and visual positioning services, thus providing convenience for speed skating training.

[0229] Corresponding to the above method embodiments, this specification also provides embodiments of a three-dimensional spatial map positioning device. Figure 6 A schematic diagram of a three-dimensional spatial map positioning device according to one embodiment of this specification is shown. Figure 6 As shown, this device is used in a server and includes:

[0230] The generation module 602 is configured to receive environmental basic data information uploaded by the acquisition device and generate a three-dimensional spatial map of the target environment based on the environmental basic data information.

[0231] The first receiving module 604 is configured to receive multimedia data of the target environment uploaded by the user equipment based on the extended reality framework;

[0232] The determination module 606 is configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user equipment.

[0233] Optionally, the generation module 602 is further configured to:

[0234] The environmental basic data information is preprocessed to obtain an initial three-dimensional spatial map of the target environment;

[0235] The three-dimensional spatial map is determined based on the initial three-dimensional spatial map, wherein the accuracy of the three-dimensional spatial map is higher than that of the initial three-dimensional spatial map.

[0236] Optionally, the generation module 602 is further configured to:

[0237] The accumulated errors in the initial 3D spatial map are eliminated using a loop closure detection method to obtain the 3D spatial map.

[0238] Optionally, the device further includes:

[0239] The second receiving module is configured to receive the location annotation data of the target environment;

[0240] The annotation module is configured to annotate the three-dimensional spatial map based on the location annotation data.

[0241] Optionally, the second receiving module is further configured to:

[0242] Receive a planar map of the target environment;

[0243] Accordingly, the annotation module is further configured as follows:

[0244] Obtain location data from the planar map, and annotate the three-dimensional spatial map based on the location data from the planar map.

[0245] Optionally, the device further includes:

[0246] The acquisition module is configured to acquire the network location information of the user equipment;

[0247] The sending module is configured to determine the target edge server based on the network location information and send the three-dimensional spatial map to the target edge server.

[0248] The positioning device for a three-dimensional spatial map provided in this specification, applied to a server, includes: a generation module configured to receive environmental basic data information uploaded by a data acquisition device and generate a three-dimensional spatial map of a target environment based on the environmental basic data information; a first receiving module configured to receive multimedia data of the target environment uploaded by a user device based on an extended reality framework; and a determination module configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user device.

[0249] By receiving multi-dimensional environmental data uploaded by acquisition devices and constructing a 3D spatial map of the target environment using environmental data from different dimensions, the accuracy of the 3D spatial map can be improved. The interaction between user devices and the server is achieved through an extended reality framework, enabling broad compatibility of the 3D spatial map's positioning system and resolving issues of low terminal compatibility and adaptation difficulties. Visual positioning services deployed on the server enable positioning within the 3D spatial map. Coupled use of the visual positioning service with the extended reality framework further enhances the accuracy of 3D spatial positioning, and during the positioning process, no further deployment of other conditions in the target environment is required, significantly reducing costs. The server can also determine the target edge server based on the user device's network location information. After generating the 3D spatial map, the target edge server interacts with the user device, thereby reducing network latency and improving response speed.

[0250] The above is a schematic scheme of a three-dimensional spatial map positioning device according to this embodiment. It should be noted that the technical solution of this three-dimensional spatial map positioning device and the technical solution of the three-dimensional spatial map positioning method described above belong to the same concept. For details not described in detail in the technical solution of the three-dimensional spatial map positioning device, please refer to the description of the technical solution of the three-dimensional spatial map positioning method described above.

[0251] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0252] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0253] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0254] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 700 can also be a mobile or stationary server.

[0255] The processor 720 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0256] The above is a schematic representation of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned three-dimensional spatial map positioning method belong to the same concept. Details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the aforementioned three-dimensional spatial map positioning method.

[0257] This specification also provides an embodiment of an augmented reality (AR) device or a virtual reality (VR) device, including:

[0258] Memory, processor, and display;

[0259] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they perform the following steps:

[0260] Acquire multimedia data of the target environment and send the multimedia data to the server based on the extended reality framework;

[0261] Receive location information sent by the server, and determine the user's current location based on the location information;

[0262] Generate an AR environment image based on the user's current location; or

[0263] Based on the user's current location, corresponding reference information is obtained, wherein the reference information includes preset reference information and / or real-time reference information. The reference information is information for the user's reference. The preset reference information is reference information pre-entered in the user's device, and the real-time reference information is reference information displayed in real time according to the user's real-time location.

[0264] The AR environment image or the reference information is rendered onto the display of the augmented reality (AR) device or the virtual reality (VR) device for display.

[0265] The above is an illustrative scheme of an augmented reality (AR) device or a virtual reality (VR) device according to this embodiment. It should be noted that the technical solution of this AR device or VR device belongs to the same concept as the technical solution of the aforementioned three-dimensional spatial map positioning method. Details not described in detail in the AR device or VR device technical solution can be found in the description of the aforementioned three-dimensional spatial map positioning method.

[0266] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described three-dimensional spatial map positioning method.

[0267] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-described three-dimensional spatial map positioning method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described three-dimensional spatial map positioning method.

[0268] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described three-dimensional spatial map positioning method.

[0269] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above-described three-dimensional spatial map positioning method belong to the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-described three-dimensional spatial map positioning method.

[0270] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0271] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0272] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0273] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0274] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A positioning system for a three-dimensional spatial map, comprising: Data acquisition equipment, user equipment, and servers; The acquisition device is configured to acquire basic environmental data information of the target environment and send the basic environmental data information to the server, wherein the basic environmental data information includes multi-dimensional sensor data information; The server is configured to receive the environmental basic data information and generate a three-dimensional spatial map of the target environment based on the environmental basic data information. The user equipment is configured to acquire multimedia data of the target environment and send the multimedia data to the server based on an extended reality framework; The server is also configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user device; The server is further configured to perform image retrieval in several small maps based on multimedia data of the target environment, and to determine a target small map based on the similarity between the multimedia data and each small map, wherein the small map includes a panoramic view of the target environment and views from different angles; and to perform feature matching between each frame of multimedia data image and the target small map to determine the user's location information.

2. The system of claim 1, wherein the acquisition device is further configured to acquire the environmental basic data information through a sensor aggregation platform operating in the acquisition device, wherein, The sensor platform includes at least two of the following: a vision sensor, a laser sensor, an inertial measurement unit, and a global positioning system.

3. The system as described in claim 1, wherein the server is further configured to perform data preprocessing on the environmental basic data information to obtain an initial three-dimensional spatial map of the target environment, and to determine the three-dimensional spatial map based on the initial three-dimensional spatial map, wherein, The accuracy of the three-dimensional spatial map is higher than that of the initial three-dimensional spatial map.

4. The system of claim 3, wherein the server is further configured to eliminate accumulated errors in the initial three-dimensional spatial map based on a loop closure detection method to obtain the three-dimensional spatial map.

5. The system of claim 1, wherein the server is further configured to receive location annotation data of the target environment and annotate the three-dimensional spatial map based on the location annotation data.

6. The system of claim 5, wherein the server is further configured to receive an input instruction for location annotation data of the target environment, open an input interface in response to the input instruction, and receive location annotation data of the target environment based on the input interface.

7. The system of claim 5, further comprising a planar map acquisition device; The planar map acquisition device is configured to acquire a planar map corresponding to the target environment and send the planar map to the server, wherein... The planar map carries location data; The server is also configured to receive the planar map, obtain location data in the planar map, and annotate the three-dimensional spatial map based on the location data of the planar map.

8. The system of claim 1, wherein the user equipment is further configured to invoke the image acquisition unit to acquire image data of the target environment, wherein, The image acquisition unit is configured in the user equipment.

9. The system of claim 1, wherein the user equipment is further configured to determine the user's current location based on the location information, receive an input instruction, determine a target location in response to the input instruction, determine a navigation path based on the target location and the current location, and generate and display an AR navigation image based on the navigation path.

10. The system of claim 1, wherein the server comprises a cloud server and multiple edge servers; The cloud server is configured to obtain the network location information of the user device, determine a target edge server from the plurality of edge servers based on the network location information, and send the three-dimensional spatial map to the target edge server; The target edge server is configured to receive and store the three-dimensional spatial map; The user equipment is also configured to acquire multimedia data of the target environment and send the multimedia data to the target edge server based on the extended reality framework; The target edge server is also configured to determine the user's location information based on the three-dimensional spatial map and the multimedia data, and send the location information to the user device.

11. The system of claim 1, wherein the user equipment is further configured to receive the location information, determine the user's current location based on the location information, and obtain and display corresponding reference information according to the user's current location, wherein, The reference information includes preset reference information and / or real-time reference information. The reference information is information for user reference. The preset reference information is reference information pre-entered into the user device, and the real-time reference information is reference information displayed in real time according to the user's real-time location.

12. The system according to any one of claims 1-11, further comprising a data analysis device; The user equipment is further configured to acquire the user's data information to be analyzed and send the data information to be analyzed to the data analysis device; The data analysis device is configured to perform data analysis on the data to be analyzed and obtain data analysis results.

13. The system of claim 12, further comprising: The user equipment is further configured to acquire the user's gliding data information and send the gliding data information to the data analysis device, wherein the gliding data information is training data generated during gliding training; The data analysis device is configured to perform data analysis on the gliding data information, and obtain and display the user's gliding data analysis results.

14. A positioning method for a three-dimensional spatial map, applied to a server, comprising: Receive basic environmental data information uploaded by the acquisition device, and generate a three-dimensional spatial map of the target environment based on the basic environmental data information; Receive multimedia data of the target environment uploaded by the user equipment based on the extended reality framework; The user's location information is determined based on the 3D spatial map and the multimedia data, and the location information is sent to the user device. The determination of the user's location information based on the 3D spatial map and the multimedia data includes: performing image retrieval in several small maps based on the multimedia data of the target environment, and determining a target small map based on the similarity between the multimedia data and each small map, wherein the small map includes a panoramic view of the target environment and views from different angles; and performing feature matching between each frame of multimedia data image and the target small map to determine the user's location information.

15. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the positioning method for the three-dimensional spatial map of claim 14.

16. An augmented reality (AR) device or a virtual reality (VR) device, comprising: Memory, processor, and display; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they perform the following steps: Acquire multimedia data of the target environment and send the multimedia data to the server based on the extended reality framework; The system receives location information sent by the server and determines the user's current location based on the location information. The location information is obtained by the server performing image retrieval on several small maps based on multimedia data of the target environment and determining the target small map based on the similarity between the multimedia data and each small map. Each frame of multimedia data image is matched with the target small map to determine the location. The small map includes a panoramic view of the target environment and views from different angles. Generate an AR environment image based on the user's current location; or Based on the user's current location, corresponding reference information is obtained, wherein the reference information includes preset reference information and / or real-time reference information. The reference information is information for the user's reference. The preset reference information is reference information pre-entered in the augmented reality (AR) device or the virtual reality (VR) device, and the real-time reference information is reference information displayed in real time according to the user's real-time location. The AR environment image or the reference information is rendered onto the display of the augmented reality (AR) device or the virtual reality (VR) device for display.

Images