Large space ar experience method, system, electronic device, and storage medium

Abstract

The application provides a large-space AR experience method, system, electronic equipment and storage medium, the method comprises the following steps: an AR terminal runs a SLAM system; the AR terminal requests positioning from a VPS at a frequency of p Hz; the AR terminal receives a positioning result and a scene type SxLy, and fuses the SLAM system to perform large-space AR experience; the AR terminal adjusts the frequency of requesting the VPS to Qn Hz according to the scene type SxLy, and re-requests positioning from the VPS. The VPS outputs a current scene type SxLy while outputting a 6Dof pose, and the AR terminal dynamically adjusts the frequency of requesting the VPS in the next period of time according to the scene type, and applies the 6Dof result, so that the VPS and the SLAM use different positioning frequencies when coping with different environments and scenes, and the 6DOF pose provided by the VPS provides stable large-space AR experience in a complex and changeable scene.

Description

Technical Field

[0001] This invention belongs to the field of AR terminal positioning technology, specifically relating to large-space AR experience methods, systems, electronic devices, and storage media. Background Technology

[0002] Definition of SLAM: SLAM stands for Simultaneous Localization and Mapping, a research field focusing on localization and mapping technologies. In existing technologies, the computing units (PC computing units and mobile computing units) used by VPS and on-device SLAM differ in scale; the available data also differs. VPS generally only uses visual information from images acquired on-device, while on-device SLAM requires both on-device image data and high-frequency sensor data. Therefore, VPS and SLAM have their advantages and disadvantages in dealing with different environments and scenarios. This results in traditional SLAM combined with 6DOF pose correction provided by a fixed-time-frequency VPS being unable to provide a stable large-space AR experience in complex and ever-changing scenarios. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a large-space AR experience method, system, electronic device, and storage medium. This addresses the issue that existing technologies such as VPS and SLAM have their own advantages and disadvantages in dealing with different environments and scenarios, resulting in the inability of traditional SLAM combined with 6DOF pose correction provided by a fixed-time-frequency VPS to provide a stable large-space AR experience in complex and ever-changing scenarios.

[0004] One embodiment of the present invention provides a method for large-space AR experience, including: Step S100: The AR terminal runs the SLAM system; Step S200: The AR terminal requests 6DoF positioning from the VPS at pHz; Step S300: The AR terminal receives the 6DoF positioning result and scene type SxLy, and integrates the SLAM system to perform a large-space AR experience; Step S400: The AR terminal adjusts the frequency of requesting the VPS to Qn Hz according to the scene type SxLy, and re-requests positioning from the VPS; In step S500, the VPS repositions itself according to the Qn Hertz corresponding to the scene type SxLy, enabling the AR terminal to experience large-space AR more stably.

[0005] In one embodiment, In step S200, p Hertz is a fixed value.

[0006] In one embodiment, in step S300, the scene type SxLy includes: The scene type based on the current environment of the AR terminal is denoted as Sx; Based on the depth of field of the AR terminal camera's image of the surrounding environment, it is divided into shallow depth of field and large depth of field, denoted as Ly. The scene type and depth of field are arranged and combined to form different scenes, denoted as SxLy.

[0007] In one embodiment, in Sx, x is 1, 2, 3, 4, 5, 6 or 7, corresponding to multiple scene types, including: default scene, shopping mall, square, natural landscape, crowd, riverside or city.

[0008] In one embodiment, in Ly, y is 1 or 2, where 1 represents shallow depth of field and 2 represents large depth of field.

[0009] In one embodiment, in step S400, for multiple different scenarios, each scenario corresponds to a positioning frequency Qn Hz. While outputting the 6DoF pose, the VPS introduces a scene understanding algorithm to output the current scene type SxLy, which corresponds to multiple different scenarios, each scenario corresponds to a positioning frequency Qn Hz. The AR terminal dynamically adjusts the frequency of requesting the VPS for the next period of time according to the scene type, and applies the 6DoF result to SLAM tracking.

[0010] In one embodiment, the shallow depth of field is less than 10 meters, and the deep depth of field is greater than 10 meters.

[0011] One embodiment of the present invention also provides a large-space AR experience system, applicable to the large-space AR experience method as described in any of the above embodiments, including: The AR terminal requests location from the VPS based on the scene type SxLy; VPS is a system service that provides high-precision 6DoF data for AR terminals.

[0012] One embodiment of the present invention also provides an electronic device, including a processor and a storage medium storing a computer program, which, when executed by the processor, implements the large-space AR experience method as described in any of the above embodiments.

[0013] One embodiment of the present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the large-space AR experience method as described in any of the above embodiments.

[0014] The large-space AR experience method or system provided in the above embodiments has the following beneficial effects: Based on the combination of scene types SxLy, and through different positioning frequencies corresponding to different scene type combinations, the VPS, while outputting the 6DOF pose, introduces a scene understanding algorithm to output the current scene type SxLy. The AR terminal dynamically adjusts the frequency of requesting the VPS over the next period of time according to the scene type and applies the 6DOF result. This allows the VPS and SLAM to use different positioning frequencies when dealing with different environments and scenes, enabling the 6DOF pose correction provided by the VPS to provide a stable large-space AR experience in complex and ever-changing scenes. By dynamically adjusting the positioning frequency, the strategy compensates for the low robustness of the technology, allowing the AR terminal to experience large-space AR more stably. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0016] Figure 1 This is an overall flowchart of the large-space AR experience method in the embodiments of this application; Figure 2 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application; Figure 3 The dataset provided for training the scene classifier in the embodiments of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indicators will also change accordingly.

[0019] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0020] One embodiment of the present invention provides a method for large-space AR experience, including: Step S100: The AR terminal runs the SLAM system; Step S200: The AR terminal requests 6DoF positioning from the VPS at pHz (pHz is a fixed value of 6); Step S300: The AR terminal receives the 6DoF positioning result and scene type SxLy, and integrates the SLAM system to perform a large-space AR experience; Step S400: The AR terminal adjusts the frequency of requesting the VPS to Qn Hz according to the scene type SxLy, and re-requests positioning from the VPS; In step S500, the VPS repositions itself according to the Qn Hertz corresponding to the scene type SxLy, enabling the AR terminal to experience large-space AR more stably.

[0021] Common AR terminal devices include all hardware devices that can run augmented reality technology, including smartphones, AR glasses, AR projectors, and in-vehicle AR HUDs. They all have a series of environmental detection sensors such as depth cameras and color cameras, which enable them to recognize gestures, 3D objects, planes, images, and so on, and register virtual information (text, images, 3D models, video, sound, etc.) to the target location in the real world. VPS, or Visual Positioning System, is a system service that uses computer vision technology as its core. It provides high-precision 6DoF data to terminal devices by taking image data captured by known terminal devices as input, thereby estimating the position and orientation of the terminal devices. SLAM: Simultaneous Localization and Mapping. In an unknown environment, a robot (including AR glasses, smartphones and other mobile AR terminals) starts moving from an unknown location. During the movement, it performs self-localization based on its location and map, and builds an incremental map based on its self-localization, so as to realize the robot's autonomous localization and navigation. Large-space AR technology, also known as World AR (proposed by Google) and Spatial AR (proposed by Baidu), aims to combine the 6DoF high-precision pose of a VPS with the SLAM tracking and positioning system on the AR terminal to enable AR experiences in any spatial scenario. In general, large-space AR relies on the VPS providing 6DoF information at a fixed frequency (e.g., 5 seconds per second) to the SLAM on the AR terminal for tracking error correction, achieving a high-precision AR experience.

[0022] The method for AR terminal devices to acquire the required image data includes the following steps: Step S1: Collect two-dimensional image data around the AR terminal; Step S2: Recognize the two-dimensional image data using a scene understanding method and output the scene type corresponding to the two-dimensional image; Step S3: Collect depth sensor data on the AR terminal and output the depth type; Step S4: Combine the results of Step S2 and Step S3 to output the permutation and combination results of scene type and depth of field type, and then output the VPS request frequency. The client runs large-space AR according to the VPS request frequency. Step S2 is not limited to a specific algorithm aimed at scene understanding. The implementation process generally includes the following four steps: P1: Determine the target scene type (including: shopping mall, square, natural landscape, crowd, riverside, urban area, etc.); P2: Collect scene type image data; P3: Train the classifier; P4: Input image data to perform scene classification and recognition and output results. Step P4 in step S2 includes performing a recognition operation on the server and the AR terminal computing unit; In step S3, the depth data output is classified into the corresponding depth type. The depth type can be customized according to the numerical range based on its size. Among them, the final result Result generated by the permutation and combination of the results in steps S2 and S3 supports custom configuration of the VPS location frequency corresponding to the result. The VPS frequency configuration on the AR terminal includes the following configurations: 1. After each Result is received, the program runs immediately according to the corresponding VPS positioning frequency; 2. After each Result is received, the program runs at a fixed VPS positioning frequency for the next number of VPS requests until the number of VPS requests ends, and then the program runs at the VPS positioning frequency corresponding to the next Result. In this embodiment, based on the combination of scene types SxLy, and through different positioning frequencies corresponding to different scene type combinations, the VPS, while outputting the 6DOF pose, incorporates a scene understanding algorithm to output the current scene type SxLy. The AR terminal dynamically adjusts the frequency of requesting the VPS over a period of time based on the scene type and applies the 6DOF results. This allows the VPS and SLAM to use different positioning frequencies when dealing with different environments and scenes, enabling the 6DOF pose correction provided by the VPS to offer a stable large-space AR experience in complex and ever-changing scenes. By dynamically adjusting the positioning frequency, the issue of low robustness in the technology is strategically compensated for, allowing the AR terminal to experience large-space AR more stably. Through this method, before overcoming technical bottlenecks, the issue of low robustness in the technology is compensated for by dynamically adjusting the positioning frequency, thereby allowing the AR terminal to experience large-space AR technology applications more stably and improving the user experience.

[0023] In one embodiment, in step S300, The scene type based on the current environment of the AR terminal is denoted as Sx (x is 1, 2, 3, 4, 5, 6 or 7, corresponding to multiple scene types, including: default scene, shopping mall, square, natural landscape, crowd, riverside or city). The depth of field of the AR terminal camera is divided into small depth of field and large depth of field, denoted as Ly (y is 1 or 2, where 1 is small depth of field and 2 is large depth of field). The small depth of field is less than 10 meters and the large depth of field is greater than 10 meters. The scene types and depth of field sizes are combined to form different scenes, denoted as SxLy, corresponding to multiple different scenes. Each scene corresponds to a positioning frequency Qn Hz, and the corresponding results are shown in the table below:

[0024] As shown in the table, assuming the output results x=1, y=2, i.e. S1L2=2, the AR terminal dynamically adjusts the frequency of requesting VPS to 2 Hz for a period of time according to the scene type, and applies the 6DoF results to SLAM tracking. In step S400, while outputting the 6DoF pose, the VPS introduces a scene understanding algorithm to output the current scene type SxLy, which corresponds to multiple different scenes. Each scene corresponds to a positioning frequency Qn Hertz. The AR terminal dynamically adjusts the frequency of requesting the VPS for the next period of time based on the scene type (assuming that the scene type returned each time is the same, then Qn should always be the same, and "the period of time" should be the time during which Qn remains unchanged), and applies the 6DoF result to SLAM tracking.

[0025] In this embodiment, the dynamic adjustment of the positioning frequency is used to compensate for the low robustness of the technology, thereby enabling AR terminals to experience large-space AR more stably.

[0026] One embodiment of the present invention also provides a large-space AR experience system, applicable to the large-space AR experience method as described in any of the above embodiments, including: The AR terminal requests location from the VPS based on the scene type SxLy; VPS provides system services for AR terminals to provide high-precision 6DoF data. The SLAM system is installed within the AR terminal.

[0027] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a large-space AR experience method. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0028] In one embodiment, an electronic device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 2As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a large-space AR experience method, and the database stores data.

[0029] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0030] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0031] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for large-space AR experience, characterized in that, include: Step S100: The AR terminal runs the SLAM system; Step S200: The AR terminal requests 6DoF positioning from the VPS at pHz; Step S300: While outputting the 6DoF pose, the VPS introduces a scene understanding algorithm to output the current scene type SxLy. The implementation process of the scene understanding algorithm includes determining the target scene type, collecting scene type image data, training a classifier, inputting image data to perform scene classification and recognition, and outputting the results. The scene type SxLy includes a scene type Sx based on the current environment of the AR terminal and a Ly based on the depth of field size of the image captured by the AR terminal's camera. The scene type and depth of field size are arranged and combined into different scenes, which are denoted as SxLy. In Sx, x is 1, 2, 3, 4, 5, 6 or 7, corresponding to the default scene, shopping mall, square, natural landscape, crowd, riverside or city. In Ly, y is 1 or 2, where 1 is small depth of field and 2 is large depth of field. The AR terminal receives the 6DoF positioning results and the scene type SxLy, and integrates the SLAM system to perform a large-space AR experience. Step S400: The AR terminal adjusts the frequency of requesting the VPS to Qn Hz according to the scene type SxLy, and re-requests positioning from the VPS; In step S500, the VPS repositions itself according to the Qn Hertz corresponding to the scene type SxLy, enabling the AR terminal to experience large-space AR more stably.

2. The large-space AR experience method as described in claim 1, characterized in that, In step S200, p Hertz is a fixed value.

3. The large-space AR experience method as described in claim 1, characterized in that, In step S400, while outputting the 6DoF pose, the VPS introduces a scene understanding algorithm to output the current scene type SxLy, which corresponds to multiple different scenes. Each scene corresponds to a positioning frequency Qn Hertz. The AR terminal dynamically adjusts the frequency of requesting the VPS in the next period of time according to the scene type, and applies the 6DoF result to SLAM tracking.

4. The large-space AR experience method as described in claim 1, characterized in that, The shallow depth of field is less than 10 meters, and the deep depth of field is greater than 10 meters.

5. A large-space AR experience system, applicable to the large-space AR experience method as described in any one of claims 1-4, characterized in that, include: The AR terminal requests location from the VPS based on the scene type SxLy; VPS is a system service that provides high-precision 6DoF data for AR terminals.

6. An electronic device comprising a processor and a storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large-space AR experience method as described in any one of claims 1-4.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the large-space AR experience method as described in any one of claims 1-4.

Images