Method and device for presenting ar virtual information

Abstract

The present application aims to provide a method and device for presenting AR virtual information. The method specifically comprises: acquiring the current electronic fence record of the current electronic fence where a corresponding user equipment is located, wherein the current electronic fence record comprises fence identification information of the current electronic fence, positioning information and one or more pieces of AR virtual information, and the AR virtual information comprises AR virtual content and corresponding position information; capturing a real-time scene image of the current scene by means of a corresponding photographic apparatus, and on the basis of the real-time scene image and the positioning information, determining corresponding real-time pose information; and on the basis of the position information of the one or more pieces of AR virtual information included in the current electronic fence record, and the real-time pose information, superimposing and presenting the AR virtual content of the one or more pieces of AR virtual information. In the present solution, the process of acquiring real-time pose information is more comprehensive and accurate, such that more precise and reliable positioning or navigation services, etc., can be provided for users, thereby greatly improving the use experience of the users.

Description

Method and device for presenting AR virtual information

[0001] This application is based on and claims priority to an application with CN application number 202410238882.9 and application date 2024.03.01. The disclosed content of the CN application is hereby incorporated into this application as a whole. Technical Field

[0002] The present application relates to the field of communications, and in particular to a technology for presenting AR virtual information. Background Art

[0003] Positioning and navigation technologies are crucial in modern society, widely used in various fields, including transportation navigation, logistics management, smart homes, and augmented reality. Early positioning and navigation technologies relied primarily on the Global Positioning System (GPS). However, in environments such as indoors, in urban canyons, and within tall buildings, GPS signals can be obstructed and interfered with by multipath effects, resulting in reduced positioning accuracy or even unavailability.

[0004] Summary of the Invention

[0005] One objective of the present application is to provide a method and device for presenting AR virtual information.

[0006] According to one aspect of the present application, a method for presenting AR virtual information is provided, wherein the method includes:

[0007] Obtaining a current geo-fence record corresponding to a current geo-fence where the user device is located, wherein the current geo-fence record includes fence identification information, positioning information, and one or more AR virtual information of the current geo-fence, wherein the AR virtual information includes AR virtual content and corresponding location information;

[0008] Capturing a real-time scene image of the current scene by a corresponding camera device, and determining corresponding real-time posture information according to the real-time scene image and the positioning information;

[0009] The AR virtual content of the one or more AR virtual information is superimposed and presented according to the position information of the one or more AR virtual information included in the current electronic fence record and the real-time posture information.

[0010] According to another aspect of the present application, a device for presenting AR virtual information is provided, wherein the device includes:

[0011] A module configured to obtain a current geo-fence record corresponding to a current geo-fence where a user device is located, wherein the current geo-fence record includes fence identification information, positioning information, and one or more AR virtual information of the current geo-fence, wherein the AR virtual information includes AR virtual content and corresponding location information;

[0012] Module one or two is used to capture a real-time scene image of the current scene through a corresponding camera device, and determine corresponding real-time posture information based on the real-time scene image and the positioning information;

[0013] The first and third modules are configured to overlay and present the AR virtual content of the one or more AR virtual information based on the location information of the one or more AR virtual information contained in the current electronic fence record and the real-time posture information.

[0014] According to one aspect of the present application, a computer device is provided, wherein the device includes:

[0015] processor; and

[0016] A memory arranged to store computer executable instructions, which when executed cause the processor to perform the steps of any of the methods described above.

[0017] According to one aspect of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed, the system performs the steps of any of the methods described above.

[0018] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction implements the steps of any of the above methods when executed by a processor.

[0019] Compared with the existing technology, this application can present one or more corresponding AR virtual information based on the electronic fence records and real-time scene image superposition. The process of obtaining real-time posture information of this solution is more comprehensive and accurate, and can provide users with more accurate and reliable positioning or navigation services, etc., greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] FIG1 shows a flow chart of a method for presenting AR virtual information according to one embodiment of the present application;

[0022] FIG2 shows a device structure diagram of a computer device according to another embodiment of the present application;

[0023] FIG3 illustrates an exemplary system that may be used to implement various embodiments described herein.

[0024] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0025] The present application is described in further detail below with reference to the accompanying drawings.

[0026] In a typical configuration of the present application, the terminal, the device of the service network, and the trusted party each include one or more processors (eg, a central processing unit (CPU), an input / output interface, a network interface, and a memory).

[0027] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of a computer-readable medium.

[0028] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0029] The devices referred to in this application include but are not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include but are not limited to any mobile electronic product that can interact with a user (for example, through a touchpad), such as a smartphone, a tablet computer, smart glasses, etc. The mobile electronic product can use any operating system, such as the Android operating system, the iOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc. The network device includes but is not limited to a computer, a network host, a single network server, a set of multiple network servers, or a cloud composed of multiple servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual supercomputer composed of a group of loosely coupled computers. The network includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless self-organizing network (Ad Hoc network), etc. Preferably, the device may also be a program running on the user device, the network device, or a device formed by integrating the user device and the network device, the network device and the touch terminal, or the network device and the touch terminal via a network.

[0030] Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or future devices that are applicable to this application should also be included in the scope of protection of this application and are included here by reference.

[0031] In the description of the present application, “plurality” means two or more, unless otherwise clearly defined.

[0032] FIG1 illustrates a method for presenting AR virtual information according to one aspect of the present application, wherein the method is applied to a computer device and primarily includes steps S101, S102, and S103. In step S101, a current geo-fence record corresponding to the current geo-fence in which a user device is located is obtained, wherein the current geo-fence record includes fence identification information, location information, and one or more AR virtual information of the current geo-fence, wherein the AR virtual information includes AR virtual content and corresponding location information, thereby obtaining the location information of the current geo-fence record. In step S102, a real-time scene image of the current scene is captured by a corresponding camera device, and corresponding real-time pose information is determined based on the real-time scene image and the location information. In step S103, AR virtual content of the one or more AR virtual information is superimposed and presented based on the location information and real-time pose information of the one or more AR virtual information contained in the current geo-fence record. The computer device includes, but is not limited to, a user device, a network device, or an integrated device of a user device and a network device. The user device includes, but is not limited to, any mobile electronic product capable of human-computer interaction with a user, such as a smartphone, tablet computer, smart glasses, smart helmet, etc. The network device includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud consisting of multiple servers. For example, a user holds and wears smart glasses, and a corresponding application, plug-in, or applet is installed on the smart glasses. By activating the corresponding application, plug-in, or applet, the smart glasses can provide the user with AR virtual information corresponding to the current scene, for example, by overlaying the corresponding AR virtual information on the current display device, or overlaying the corresponding AR virtual information on the current real-time scene image.

[0033] For example, the electronic fence refers to a virtual fence, which is different from the component structure of the physical fence detection in the traditional sense. The electronic fence is used to represent the range of a specific spatial position in the map. Specifically, combined with actual usage needs, the fence area range of the electronic fence can be a spatial position based on a certain plane area as a reference plane, including any height. For example, after the corresponding plane area is selected on the ground in a 2D or 3D map, the corresponding spatial point at any altitude of the plane area can be determined to be within the electronic fence; or the fence area range of the electronic fence can be any space including a three-axis range, for example, a closed space composed of multiple planes or curved surfaces, specifically, a closed space defined by the spatial boundaries of one or more rooms on the Xth floor of a high-rise building, etc. The electronic fence record includes the fence identification information of the electronic fence, and the positioning information and AR virtual information within the allowed range of the electronic fence, etc.; wherein, the fence identification information is used to indicate the unique identification of the electronic fence, and the positioning information includes positioning data for calculating the camera posture information of the smart glasses or indication information of the corresponding positioning data, etc., specifically, for example, sparse point cloud map information and / or 2D feature information within the fence area of ​​the electronic fence, or indication information of the positioning data for indicating sparse point cloud map information and / or 2D feature information, etc., and sparse point cloud map information and / or 2D feature information can be further obtained based on the indication information. The AR virtual information includes annotation information used to indicate a point, line, surface, object or area in the corresponding scene. Each AR virtual information includes virtual content information, for example, pictures, videos, 3D models, PDF files, office documents, form information, audio, hyperlinks, application call information (used to execute relevant instructions of the application, etc.), real-time sensor information (used to connect to sensors and obtain sensor data of target objects), graffiti or other digital elements and other annotation information added to a certain point, line, surface, object or area. Of course, those skilled in the art should understand that the above-mentioned annotation information is only for example and is not limited.

[0034] One or more electronic fence records corresponding to the current scene can be stored in the smart glasses terminal, or stored in the server side, etc. The smart glasses terminal can call / load part or all of the electronic fence records in at least one electronic fence record stored locally, or the smart glasses send a request to the server and receive part or all of the electronic fence records in at least one electronic fence record sent by the server side.

[0035] In some embodiments, the method further includes step S104 (not shown), in which a scene image of the current scene is obtained, and the positioning information and corresponding scene map of the current scene are determined based on the scene image; a user operation of any user in the scene map of the current scene is obtained, and one or more AR virtual information of the current scene, and fence identification information and fence areas of one or more electronic fences are determined based on the user operation; one or more electronic fence records of the current scene are generated based on the fence identification information, fence areas, one or more AR virtual information, and the positioning information of the one or more electronic fences, wherein each electronic fence record includes the fence identification information, positioning information and corresponding at least one AR virtual information of the electronic fence, and the AR virtual information includes AR virtual content and corresponding location information. For example, the scene image of the current scene can be scanned by a camera device, so as to determine the corresponding positioning information and scene map based on the scene image. A computer device (which may be different from a device presenting AR virtual information) obtains scene image information in advance by scanning the real environment, such as using a panoramic camera, lidar or ordinary camera for shooting. The computer device can perform data processing on the scene image of the current scene to obtain the overall sparse point cloud map information (positioning information of the current scene) and texture map information (scene map) corresponding to the scene. For example, a sparse map is constructed first, and then a texture map is constructed. The construction of the sparse map can be based on the existing incremental motion structure recovery technology (SFM), and the construction of the texture map can be based on the traditional multi-view stereo vision technology (MVS). The construction of the sparse map and the texture map is only for example and not limited here. After the computer device obtains the overall sparse point cloud map and texture map of the current scene, it can manage the electronic fence and / or AR virtual information of the current scene (such as adding, modifying, deleting, etc.) based on the overall sparse point cloud map and / or texture map. For example, based on any user's editing operation on the overall sparse point cloud map and / or texture map, one or more corresponding electronic fences and / or AR virtual information are generated, and then the electronic fence record of the current scene is established based on the electronic fence (including the positioning information corresponding to the electronic fence) and / or AR virtual information. In some cases, there are corresponding overall sparse point cloud maps and texture maps for the corresponding scenes. When the user edits the fence area in the overall texture map, the correspondence between the sparse point cloud map and the texture map can be used to determine the target sparse area of ​​the fence area in the overall sparse point cloud map based on the fence area, and the target sparse area can be determined as the sparse point cloud map information of the fence area of ​​the electronic fence being edited.That is, after the computer device completes the editing of the annotation of the fence area in the scene on the overall texture map (such as the determination of the fence area and the fence identification information of the fence area, etc.), it can obtain the sparse point cloud map information of the fence area on the overall sparse point cloud map. Furthermore, the sparse point cloud map information of the fence area is used as the positioning information corresponding to the electronic fence. Similarly, after the user device completes the editing of the annotation of the fence area in the scene on the overall sparse point cloud map, it can use the sparse point cloud map information corresponding to the fence area in the overall sparse point cloud map as the positioning information of the electronic fence. In some embodiments, any user performs operations such as box selection on the overall sparse point cloud map and / or texture map corresponding to the current scene to generate a corresponding electronic fence, wherein the target area includes but is not limited to a box, a circle, other polygons, or a custom polygon composed of multiple points selected by the user. For example, the corresponding user operation interface includes an area selection method of a preset shape (e.g., a box, circle, or other polygon, etc.), and the selected target area is determined based on the user's selection operation on the preset shape and based on the selection of a specific position / area of ​​the three-dimensional map, such as selecting a specific position as the center of the circle / polygon, entering or pulling a certain distance as the corresponding radius / circumscribed circle radius of the polygon, etc., thereby determining the corresponding target area, or selecting a specific position as the center of the circle / polygon, selecting one or more corner points as points on the circumference / corner points of the polygon, etc., thereby determining the corresponding target area, etc.; for example, based on the user's selection operation on multiple position points, multiple points are sequentially connected based on the selection order to form a closed pattern, thereby determining the corresponding target area, etc. In some embodiments, any user adds AR virtual information to the overall sparse point cloud map and / or texture map corresponding to the current scene, and the AR virtual information includes corresponding virtual content and location information corresponding to the virtual content (e.g., location in the sparse point cloud map or texture map, etc.). Among them, the computer device may include one or more devices, and operations such as scene image capture, map information acquisition, AR virtual information editing, and division of electronic fences can be completed independently by the corresponding same device, or can be completed by multiple different devices participating in cooperation; wherein, the user operation of editing AR virtual information and / or dividing electronic fences can include respective user operations obtained from different user devices. Among them, the fence ranges between one or more electronic fences can be separate, or partially overlap, or some electronic fences can be included in the spatial position of another electronic fence. The number of AR virtual information can also be one or more. There is no specific order in the generation process of the one or more electronic fences and AR virtual information. In some embodiments, the server side can update the electronic fences and AR virtual information in real time based on the calls of each user. In other embodiments, each user side locally updates its own updates to the electronic fences and AR virtual information.In some embodiments, after obtaining a scene map, 2D feature information may be added to an object / area / location in the scene map (e.g., added directly to the scene map, or added based on a scanned image of the scene by any user and mapped to the scene map). The 2D feature information includes feature points of a 2D identification map and corresponding location information (e.g., a location in the scene map, etc.). Furthermore, in some embodiments, the sparse point cloud map information and / or 2D feature information of the fenced area of ​​an electronic fence is used as positioning information of the electronic fence, etc. In some cases, the positioning information may also be indicative information regarding the sparse point cloud map information and / or 2D feature information of the fenced area of ​​the electronic fence, such as the location or data range of the sparse point cloud map information and / or 2D feature information. Based on the location or data range, the computer device may separate the sparse point cloud map information of the electronic fence from the overall sparse point cloud map information of the current scene, or load the sparse point cloud map information and / or 2D feature information corresponding to the electronic fence from a specific storage area.

[0036] In step S101, a current electronic fence record of the current electronic fence where the corresponding user device is located is obtained, wherein the current electronic fence record includes fence identification information, positioning information and one or more AR virtual information of the current electronic fence, and the AR virtual information includes AR virtual content and corresponding location information. For example, the smart glasses include a corresponding position sensor, and the device location of the smart glasses can be obtained through the position sensor, such as obtaining the current geographical location information of the device through the GPS system; for example, the smart glasses also include a short-range communication device (for example, Bluetooth, radar receiver, radio receiver, sound wave receiver, etc.), based on which a communication connection can be established with a beacon device pre-set in the real scene, thereby determining the connected beacon device. Based on the corresponding device location and / or the connected beacon device and the beacon device and / or GPS location corresponding to each electronic fence, if the device location is included in the GPS location range corresponding to a certain electronic fence, or if the beacon device connected to the device is included in the beacon device corresponding to a certain electronic fence, then the certain electronic fence is determined to be the current electronic fence. Based on the electronic fence record corresponding to the current electronic fence, the electronic fence record can be called / loaded to obtain the positioning information and AR virtual information corresponding to the current electronic fence. In some embodiments, there is a correspondence between the electronic fence record and the beacon device (e.g., beacon device location and / or beacon device identification information, etc.) and / or GPS location. For example, the beacon device location and / or GPS location is stored in the electronic fence record as a type of electronic fence information. For another example, the beacon device identification is stored in the electronic fence record as a type of electronic fence information. For another example, the beacon device identification, beacon device location and / or GPS location indication information is stored in the electronic fence record as a type of electronic fence information. The beacon device location, identification information and / or GPS location can be further determined by the indication information, so that the electronic fence where the user device is currently located can be located by obtaining the location of the user device (e.g., through a connected beacon device or a location obtained by GPS). If the electronic fence record corresponds to a beacon device, the electronic fence can correspond to one or more beacon devices. If the electronic fence record corresponds to a GPS location, the electronic fence can correspond to one or more groups of GPS locations, wherein each group of GPS locations can include, but is not limited to, a position range corresponding to a rectangle, a circle, other polygons, or a custom polygon composed of multiple points selected by the user, etc., which is not limited here.

[0037] In some embodiments, the fenced area of ​​the current electronic fence includes, but is not limited to, indoor areas and outdoor areas. For example, depending on the accuracy of determining the device location of the smart glasses in the real environment, different device location acquisition methods can be set for different fenced areas (e.g., outdoor areas and indoor areas). Typically, for electronic fences corresponding to indoor areas, due to weak indoor GPS signals, the device location can be accurately acquired by setting a beacon device within the electronic fence area corresponding to the indoor area. For electronic fences corresponding to outdoor areas, the device location can be acquired by setting a beacon device within the electronic fence area corresponding to the outdoor area and / or using a GPS sensor. In some cases, when the location of the device to be located includes specific floor or altitude information, the device location can also be accurately acquired by setting a beacon device at a specific floor or altitude. By setting beacons, the present application can solve the problem of user device positioning indoors and outdoors. In step S102, a real-time scene image of the current scene is captured by a corresponding camera device, and corresponding real-time posture information is determined based on the real-time scene image and the positioning information. For example, a computer device includes a corresponding camera device, which captures a real-time scene image of the current scene in which the user is currently located by activating the camera device, and determines the position and posture using the acquired positioning information, thereby acquiring real-time position and posture information of the camera device. For example, the computer device performs feature extraction on the acquired real-time scene image, matches the extracted features with sparse point cloud map information in the pre-acquired positioning information recorded by the current electronic fence, and then obtains the current position and posture of the camera device through PNP calculation. For another example, the computer device performs feature extraction on the acquired real-time scene image, matches the extracted features with 2D feature information in the pre-acquired positioning information recorded by the current electronic fence, to determine the current position and posture of the camera device.

[0038] In step S103, the AR virtual content of the one or more AR virtual information items is superimposed and presented based on the position information and real-time pose information of the one or more AR virtual information items contained in the current geo-fence record. For example, after the computer device obtains the camera pose information of the real-time scene image of the current scene, the AR virtual information in the scene can be superimposed and presented. For example, the computer device calculates the image position / display screen position of the AR virtual information in the real-time scene image / display screen based on the camera pose information and the spatial position information of the AR virtual information contained in the current geo-fence record, thereby superimposing and presenting the corresponding AR virtual information at the image position / display screen position. In some cases, the computer device may also obtain the corresponding AR virtual information based on the target area identification information and the target area location information in the real-time scene image identified by target detection, determine the location information of the corresponding area as the tag location information of the AR virtual information, and calculate the corresponding image position / display screen position based on the camera pose information and the tag location information, thereby achieving dynamic superimposition and presentation of the AR virtual information. Here, the AR virtual information superimposed and presented on the real-time scene image / display screen includes some / all of the AR virtual information in the current geo-fence record. This application determines the current electronic fence by obtaining the location of the user device, and loads the positioning information and AR virtual information corresponding to the current electronic fence. There is no need to directly load all the positioning information and AR virtual information corresponding to the scene. Without increasing the power consumption of the user device, it achieves accurate overlay of AR virtual content and improves the user experience.

[0039] In some embodiments, the positioning information includes sparse point cloud map information of the current electronic fence, and the determining of the corresponding real-time pose information based on the real-time scene image and the positioning information includes: utilizing a large-space positioning algorithm to determine the real-time pose information of the current scene based on the real-time scene image and the sparse point cloud map information. For example, the positioning information may be sparse point cloud map information within the fence range of the current electronic fence. Here, the real-time scene image may be one or more images of the current scene. We will use a single image as an example to illustrate the following embodiments. If the corresponding real-time scene image is multiple images, the processing process of each image is similar to the processing process of the single image, and the embodiments are also applicable to the processing process of multiple images. The computer device can perform scene positioning on the current scene based on the real-time scene image to determine the corresponding camera posture information, for example, by extracting features from the real-time scene image and matching them to determine the corresponding camera posture information; or, by obtaining a scene feature map of the real-time scene image, and using feature point matching and PNP calculation on the scene feature map to obtain the camera posture information of the real-time scene image, the real-time posture information includes the camera position information and camera posture information of the camera device when the corresponding real-time scene image is captured.

[0040] In some embodiments, the positioning information includes sparse point cloud map information of the current electronic fence, and the determining of the corresponding real-time pose information based on the real-time scene image and the positioning information includes: using a SLAM algorithm to determine the corresponding first real-time pose information in real time based on the real-time scene image and the sparse point cloud map information; using a large space positioning algorithm to determine the corresponding second pose information based on the real-time scene image and the sparse point cloud map information based on a preset time interval; if the second pose information is obtained, then the second pose information is determined as the real-time pose information of the current scene; if the second pose information is not obtained, then the first real-time pose information is determined as the real-time pose information of the current scene. For example, the simultaneous localization and mapping (SLAM) refers to starting to move from an unknown position in an unknown environment, positioning itself according to the position and map during the movement, and constructing an incremental map based on its own positioning to achieve autonomous positioning of the robot. When a computer device acquires images of a corresponding real-time scene, it will continuously calculate the first real-time pose information of the camera device using the SLAM algorithm. In some cases, large-space positioning technology (also known as large-space computing technology, VPS, etc.) requires a large amount of computation and is more accurate than the SLAM algorithm. To save computing resources, the corresponding real-time scene images are typically calculated selectively during the image acquisition process, such as calculating and determining the second real-time pose information of the camera device at intervals of preset image frames or at intervals of preset time (e.g., 2 seconds). In some embodiments, when the computer device acquires the corresponding second real-time pose information using the large-space positioning technology, the second real-time pose information with higher accuracy is determined as the real-time pose information of the current scene. If the corresponding second real-time pose information is not acquired, the first real-time pose information determined by the SLAM algorithm is determined as the real-time pose information of the current scene.

[0041] In some embodiments, the positioning information also includes 2D feature information contained in the electronic fence record. For example, for areas with relatively few features or unclear textures, the accuracy of the recognition results obtained by large-space positioning and / or SLAM positioning will also be affected. Based on this, corresponding 2D recognition maps can usually be set in these areas. For example, the 2D recognition map is added to a specific area / position of the scene map, and the 2D recognition feature information corresponding to the 2D recognition map is entered into the electronic fence record. The 2D feature information includes the feature points of the 2D recognition map and the position of the recognition map (for example, the position in the scene map, etc.). Specifically, the 2D recognition map is usually composed of a specific pattern or code, such as a two-dimensional barcode, QR code, AR marker, etc. These markers have unique patterns and coding structures and can be recognized and tracked by computer vision systems. In some embodiments, the positioning information corresponding to some electronic fences may include sparse point cloud map information and / or 2D feature information, etc., thereby providing users with a variety of positioning options, etc. In some embodiments, the computer device can directly determine the corresponding real-time pose information based on the 2D feature information contained in the electronic fence record; in some embodiments, the computer device can obtain multiple real-time pose information through SLAM algorithms, 2D recognition, etc., and determine the real-time pose information of the current scene from the multiple real-time pose information. For example, when the computer device obtains the corresponding second real-time pose information through large-space positioning technology, the second real-time pose information with higher accuracy is determined as the real-time pose information of the current scene; if the corresponding second real-time pose information is not obtained, it means that the features of the current real-time scene image are insufficient, and the corresponding 2D recognition algorithm and 2D feature information are called to perform 2D feature recognition. If the corresponding real-time pose information can be re-acquired through 2D feature recognition, the real-time pose information of the 2D feature recognition is determined as the real-time pose information of the current scene. In some embodiments, the method further includes step S105 (not shown). In step S105, if the calculation of the second pose information fails, the corresponding second pose information is recalculated based on the real-time scene image and the 2D feature information.

[0042] In some embodiments, the current electronic fence has a corresponding positioning algorithm; wherein determining the corresponding real-time pose information based on the real-time scene image and the positioning information includes: determining the corresponding real-time pose information based on the positioning information, the positioning algorithm, and the real-time scene image. For example, some or all of one or more electronic fences are equipped with corresponding positioning algorithms, and the positioning algorithms are used to indicate the type of positioning algorithm used by the electronic fence to calculate the camera pose information, such as a SLAM algorithm, a large-space positioning algorithm, or a 2D recognition algorithm. An electronic fence can correspond to one or more positioning algorithms. When corresponding to one positioning algorithm, it indicates that only the positioning algorithm is used to calculate the camera pose information within the electronic fence. When corresponding to multiple positioning algorithms, multiple positioning algorithms can be run according to a specific strategy to determine the final camera pose information. For example, electronic fence A corresponds to a 2D recognition algorithm, and electronic fence B corresponds to a large-space positioning algorithm and a SLAM algorithm. When the user device enters the scene corresponding to electronic fence A, it is positioned and presented with AR virtual content through the 2D recognition algorithm and the 2D feature information in the positioning information. When the user device enters the scene corresponding to electronic fence B, it is positioned in real time through the SLAM algorithm and the sparse point cloud map, and is positioned through large-space positioning at predetermined intervals. When the corresponding second real-time posture information is obtained through the large-space positioning technology, the second real-time posture information is determined as the real-time posture information of the current scene. If the corresponding second real-time posture information is not obtained, the first real-time posture information determined by the SLAM algorithm is determined as the real-time posture information of the current scene, and so on. Those skilled in the art should be able to determine that the specific algorithms and strategies here are only examples and are not limited. There is a mapping relationship between the positioning algorithm and the fence identification information of the electronic fence. The mapping relationship can be stored separately or included in the corresponding electronic fence record. After the computer device obtains the current geo-fence, it can query the geo-fence identification information of the current geo-fence and determine a preset positioning algorithm, thereby calculating the corresponding real-time pose information based on the preset positioning algorithm. In some cases, the current geo-fence may be a sub-fence of another geo-fence, and the current geo-fence may also include multiple sub-geo-fences of the current geo-fence. In this case, when the computer device determines the positioning algorithm based on the geo-fence records, it uses the positioning algorithm of the smallest geo-fence in which the device is located as the positioning algorithm for the real-time scene image.

[0043] In some embodiments, the electronic fence record corresponds to a beacon device, and the beacon device is used to broadcast a corresponding short-range communication signal. In some embodiments, the electronic fence record corresponds to a beacon device means that there is a corresponding relationship between the beacon device position and / or identification information and the electronic fence. For example, the beacon device position is stored in the electronic fence record as a kind of electronic fence information. For another example, the beacon device identification is stored in the electronic fence record as a kind of electronic fence information. For another example, the beacon device identification or the indication information of the beacon device position is stored in the electronic fence record as a kind of electronic fence information. The beacon device position or identification information can be further determined by the indication information. For example, for a specific electronic fence area in a physical scene, a corresponding beacon device is set to broadcast a short-range communication signal for the smart glasses to communicate and connect, thereby determining the device position of the smart glasses based on the communication connection. In some cases, if the smart glasses establish a short-range communication connection with a beacon device, it is determined that the smart glasses are currently within the fence range of the electronic fence corresponding to the beacon device, thereby determining the current electronic fence corresponding to the smart glasses. For example, in some embodiments, obtaining the current geo-fence record for the current geo-fence in which the user device is located includes: if the user device establishes a short-range communication connection with one of one or more beacon devices in the current scene, determining the geo-fence record corresponding to one of the one or more beacon devices as the current geo-fence record for the current geo-fence in which the user device is located. In some embodiments, the smart glasses can also determine that the smart glasses are currently within the fence range of the geo-fence corresponding to the beacon device with the greatest signal strength by obtaining signal strength information corresponding to the currently connected short-range communication signal, thereby determining the current geo-fence corresponding to the smart glasses.

[0044] In some embodiments, the fenced area includes an outdoor area; obtaining the current electronic fence record of the current electronic fence in which the user device is located includes: utilizing global positioning technology to obtain device location information corresponding to the user device; if the device location information is within the location range of a certain electronic fence, then determining the electronic fence record of the certain electronic fence as the current electronic fence record of the current electronic fence in which the user device is located. In some embodiments, the electronic fence record corresponding to GPS location information (e.g., a coordinate range consisting of longitude and latitude, etc., for example, by selecting an electronic fence area on a scene map, the GPS location information corresponding to the electronic fence area can be obtained) means that there is a corresponding relationship between the GPS location information and the electronic fence. For example, the GPS location information is stored in the electronic fence record as a form of electronic fence information. For example, indication information of the GPS location information is stored in the electronic fence record as a form of electronic fence information, and the GPS location information can be further determined by the indication information. For example, a computer device can determine the device's geographic location by receiving satellite signals corresponding to the GPS. In an open outdoor area with a good view, the geographic location information determined by the GPS signal is more accurate. Among them, if the computer device obtains that the device location information is included in the GPS location information range corresponding to a certain electronic fence, the certain electronic fence will be determined as the corresponding current electronic fence, and the electronic fence record of the certain electronic fence will be determined as the current electronic fence record of the current electronic fence where the user device is located, etc.

[0045] In some embodiments, the method further includes step S106 (not shown), in which destination location information is obtained; a corresponding virtual navigation path is generated based on the destination location information and the current location information of the user device, and the virtual navigation path is superimposed and presented based on the real-time posture information. For example, the computer device can perform virtual navigation based on a three-dimensional map of the current scene, and based on the aforementioned determined device location information and the corresponding destination location information, calculate at least one navigation path between the two, for example, through a Dijkstra algorithm, an A* algorithm, etc., and superimpose and present the corresponding virtual navigation path on the current display screen. The destination location information can be input by the user, or a recommended location recommended by the computer device to the user (for example, an adjacent parking lot, a leisure place recommendation, etc.). The virtual navigation path includes virtual prompt information for indicating the route from the current device location information to the destination location information, such as highlighted road signs and / or road labels. After the computer device determines the corresponding virtual navigation path, it can calculate the image position / display screen position of the corresponding virtual content superimposed on the real-time scene image / display screen based on the multiple virtual contents included in the virtual navigation path and the spatial position of the corresponding virtual content, combined with the real-time posture information, so as to superimpose the corresponding virtual content on the image position / screen position. In some cases, the superimposed presentation of the virtual navigation path is based on the current device position information and the virtual content within a certain range of the current field of view (for example, a visual range of 50 meters, etc.), and the like.

[0046] The above mainly introduces the various embodiments of a method for presenting AR virtual information of the present application. In addition, the present application also provides specific devices that can implement the above embodiments, which are introduced below in conjunction with Figure 2.

[0047] FIG2 shows a computer device 100 for presenting AR virtual information according to one aspect of the present application, which mainly includes a module 11 101, a module 12 102, and modules 13 103 and 14 104. Module 101 is used to obtain a current electronic fence record corresponding to the current electronic fence where the user device is located, wherein the current electronic fence record includes fence identification information, positioning information, and one or more AR virtual information of the current electronic fence, and the AR virtual information includes AR virtual content and corresponding position information; Module 102 is used to capture a real-time scene image of the current scene through a corresponding camera device, and determine corresponding real-time posture information based on the real-time scene image and the positioning information; Module 103 is used to overlay and present the AR virtual content of the one or more AR virtual information based on the position information and real-time posture information of the one or more AR virtual information contained in the current electronic fence record.

[0048] Here, the specific implementations corresponding to module 101, module 102 and module 103 shown in Figure 2 are the same or similar to the embodiments of step S101, step S102 and step S103 shown in Figure 1 above, and are therefore not repeated here and are included here by reference.

[0049] In some embodiments, the device further includes a module (not shown) configured to obtain a scene image of the current scene and determine positioning information and a corresponding scene map of the current scene based on the scene image; obtain user operations performed by any user in the scene map of the current scene, and determine one or more AR virtual information of the current scene and fence identification information and fence areas of one or more geo-fences based on the user operations; and generate one or more geo-fence records for the current scene based on the fence identification information, fence areas, the one or more AR virtual information, and the positioning information, wherein each geo-fence record includes the fence identification information, positioning information, and at least one corresponding AR virtual information of the geo-fence, the AR virtual information including AR virtual content and corresponding location information. In some embodiments, the fence areas of the current geo-fence include, but are not limited to, indoor areas and outdoor areas. In some embodiments, the positioning information includes sparse point cloud map information of the current geo-fence, and determining corresponding real-time pose information based on the real-time scene image and the positioning information includes: utilizing a large-space positioning algorithm to determine the real-time pose information of the current scene based on the real-time scene image and the sparse point cloud map information.

[0050] In some embodiments, the positioning information includes sparse point cloud map information of the current electronic fence, and the determining of the corresponding real-time pose information based on the real-time scene image and the positioning information includes: using a SLAM algorithm to determine the corresponding first real-time pose information in real time based on the real-time scene image and the sparse point cloud map information; using a large space positioning algorithm to determine the corresponding second pose information based on the real-time scene image and the sparse point cloud map information based on a preset time interval; if the second pose information is obtained, the second pose information is determined as the real-time pose information of the current scene; if the second pose information is not obtained, the first real-time pose information is determined as the real-time pose information of the current scene.

[0051] In some embodiments, the positioning information also includes 2D feature information contained in the geo-fence record. In some embodiments, the device further includes a module (not shown) for recalculating the corresponding second pose information based on the real-time scene image and the 2D feature information if the calculation of the second pose information fails.

[0052] In some embodiments, a corresponding positioning algorithm exists for the current electronic fence; wherein, determining the corresponding real-time pose information based on the real-time scene image and the positioning information includes: determining the corresponding real-time pose information based on the positioning information, the positioning algorithm, and the real-time scene image. In some embodiments, the electronic fence record corresponds to a beacon device, and the beacon device is used to broadcast a corresponding short-range communication signal. In some embodiments, obtaining the current electronic fence record of the current electronic fence where the user device is located includes: if the user device establishes a short-range communication connection with one of the one or more beacon devices in the current scene, determining the electronic fence record corresponding to one of the one or more beacon devices as the current electronic fence record of the current electronic fence where the user device is located.

[0053] In some embodiments, the fence area includes an outdoor area; obtaining the current electronic fence record of the current electronic fence where the user device is located includes: using global positioning technology to obtain the device location information of the corresponding user device; if the device location information is within the location range of a certain electronic fence, then determining the electronic fence record of the certain electronic fence as the current electronic fence record of the current electronic fence where the user device is located.

[0054] In some embodiments, the device further includes a module (not shown) for obtaining destination location information; generating a corresponding virtual navigation path based on the destination location information and the current location information of the user device, and presenting the virtual navigation path based on the real-time posture information.

[0055] Here, the specific implementations corresponding to the four modules to the six modules are the same as or similar to the embodiments of the aforementioned steps S104 to S106, and thus are not described in detail and are included herein by reference.

[0056] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium, which stores computer code. When the computer code is executed, the method described in any of the above items is executed.

[0057] The present application also provides a computer program product. When the computer program product is executed by a computer device, the method described in any one of the preceding items is executed.

[0058] The present application also provides a computer device, comprising:

[0059] one or more processors;

[0060] a memory for storing one or more computer programs;

[0061] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any one of the preceding items.

[0062] FIG3 illustrates an exemplary system that may be used to implement various embodiments described herein;

[0063] As shown in FIG3 , in some embodiments, system 300 can function as any of the aforementioned devices in the various embodiments. In some embodiments, system 300 may include one or more computer-readable media (e.g., system memory or non-volatile memory (NVM) / storage device 320) having instructions and one or more processors (e.g., processor(s) 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement the modules and thereby perform the actions described herein.

[0064] For one embodiment, system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 305 and / or any suitable device or component in communication with system control module 310 .

[0065] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0066] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. For one embodiment, system memory 315 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, system memory 315 can include Double Data Rate 4 SDRAM (DDR4 SDRAM).

[0067] For one embodiment, system control module 310 may include one or more input / output (I / O) controllers to provide interfaces to NVM / storage device 320 and communication interface(s) 325 .

[0068] For example, NVM / storage 320 may be used to store data and / or instructions. NVM / storage 320 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives).

[0069] NVM / storage device 320 may include storage resources that are physically part of the device on which system 300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 320 may be accessed over a network via communication interface(s) 325.

[0070] Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and / or with any other suitable devices. System 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.

[0071] For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 (e.g., the memory controller module 330). For one embodiment, at least one of the processor(s) 305 may be packaged together with the logic of one or more controllers of the system control module 310 to form a system in a package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310 to form a system on chip (SoC).

[0072] In various embodiments, system 300 may be, but is not limited to, a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, system 300 may have more or fewer components and / or a different architecture. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0073] It should be noted that the application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the application can be executed by a processor to realize the steps or functions described above. Similarly, the software program of the application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0074] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes but is not limited to a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0075] Communication media include media by which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media may include guided transmission media such as cables and wires (e.g., fiber optic, coaxial, etc.) and wireless (unguided transmission) media that can propagate energy waves, such as acoustic, electromagnetic, radio frequency (RF), microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data may be embodied as, for example, a modulated data signal in a wireless medium such as a carrier wave or similar mechanism such as that embodied as part of spread spectrum technology. The term "modulated data signal" refers to a signal that has one or more characteristics changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

[0076] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information / data for use by a computer system.

[0077] Here, according to one embodiment of the present application, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to run the methods and / or technical solutions based on the aforementioned multiple embodiments of the present application.

[0078] It is obvious to those skilled in the art that the present application is not limited to the details of the above-mentioned exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

Claims

1. A method for presenting AR virtual information, wherein: The method includes: Obtaining a current geo-fence record corresponding to a current geo-fence where the user device is located, wherein the current geo-fence record includes fence identification information, positioning information, and one or more AR virtual information of the current geo-fence, wherein the AR virtual information includes AR virtual content and corresponding location information; Capturing a real-time scene image of the current scene by a corresponding camera device, and determining corresponding real-time posture information according to the real-time scene image and the positioning information; The AR virtual content of the one or more AR virtual information is superimposed and presented according to the position information of the one or more AR virtual information included in the current electronic fence record and the real-time posture information.

2. The method according to claim 1, wherein The fence area of ​​the current electronic fence includes at least one of the following: indoor areas; Outdoor area.

3. The method according to claim 1, wherein The positioning information includes sparse point cloud map information of the current electronic fence; The determining corresponding real-time posture information according to the real-time scene image and the positioning information includes: A large-space positioning algorithm is used to determine the real-time pose information of the current scene based on the real-time scene image and the sparse point cloud map information.

4. The method according to claim 1, wherein The positioning information includes sparse point cloud map information of the current electronic fence, and determining corresponding real-time pose information according to the real-time scene image and the positioning information includes: Determine corresponding first real-time pose information in real time based on the real-time scene image and the sparse point cloud map information using a SLAM algorithm; Determining corresponding second pose information based on the real-time scene image and the sparse point cloud map information using a large space positioning algorithm and a preset time interval; If the second posture information is obtained, the second posture information is determined as the real-time posture information of the current scene; if the second posture information is not obtained, the first real-time posture information is determined as the real-time posture information of the current scene.

5. The method according to claim 3 or 4, wherein: The positioning information also includes 2D feature information contained in the electronic fence record.

6. The method according to claim 5, wherein: The method further comprises: If the calculation of the second pose information fails, the corresponding second pose information is recalculated according to the real-time scene image and the 2D feature information.

7. The method according to claim 1, wherein The current electronic fence has a corresponding positioning algorithm; wherein, determining the corresponding real-time posture information based on the real-time scene image and the positioning information includes: Corresponding real-time posture information is determined according to the positioning information, the positioning algorithm, and the real-time scene image.

8. The method according to claim 1, wherein The method further comprises: Acquire a scene image of the current scene, and determine positioning information of the current scene and a corresponding scene map based on the scene image; Obtaining a user operation of any user in a scene map of a current scene, and determining one or more AR virtual information of the current scene, and fence identification information and fence areas of one or more electronic fences based on the user operation; One or more electronic fence records of the current scene are generated based on the fence identification information, fence area of ​​the one or more electronic fences, the one or more AR virtual information, and the positioning information, wherein each electronic fence record includes the fence identification information, positioning information and corresponding at least one AR virtual information of the electronic fence, and the AR virtual information includes AR virtual content and corresponding location information.

9. The method according to claim 8, wherein The electronic fence records a corresponding beacon device, and the beacon device is used to broadcast a corresponding short-range communication signal.

10. The method according to claim 9, wherein: The obtaining of a current electronic fence record of the current electronic fence where the user equipment is located includes: If the user device establishes a short-range communication connection with one of the one or more beacon devices in the current scene, the electronic fence record corresponding to one of the one or more beacon devices is determined as the current electronic fence record of the current electronic fence where the user device is located.

11. The method according to claim 8, wherein The fence area includes an outdoor area; and obtaining a current electronic fence record of the current electronic fence where the user equipment is located includes: The device location information of the corresponding user device is obtained using global positioning technology. If the device location information is within the location range of a certain electronic fence, the electronic fence record of the certain electronic fence is determined as the current electronic fence record of the current electronic fence where the user device is located.

12. The method according to claim 1, wherein The method further comprises: Get the destination location information; A corresponding virtual navigation path is generated based on the destination location information and the current location information of the user device, and the virtual navigation path is superimposed and presented based on the real-time posture information.

13. A device for presenting AR virtual information, wherein: The device includes: A module configured to obtain a current geo-fence record corresponding to a current geo-fence where a user device is located, wherein the current geo-fence record includes fence identification information, positioning information, and one or more AR virtual information of the current geo-fence, wherein the AR virtual information includes AR virtual content and corresponding location information; Module one or two is used to capture a real-time scene image of the current scene through a corresponding camera device, and determine corresponding real-time posture information based on the real-time scene image and the positioning information; The first and third modules are configured to overlay and present the AR virtual content of the one or more AR virtual information based on the location information of the one or more AR virtual information contained in the current electronic fence record and the real-time posture information.

14. A computer device, wherein: The device includes: processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: The computer program / instructions, when executed, cause the system to perform the steps of the method as claimed in any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images