Offline scene information collection method, platform, system and medium based on AR terminal

Abstract

The present invention discloses an offline scene information collection method, platform, system and medium based on an AR terminal, the method comprising: receiving real-scene data collected by an AR terminal; performing scene standard detection on the real-scene data to confirm whether the real-scene data meets the preset collection standard; performing scene image and text recognition on the real-scene data when the real-scene data meets the preset collection standard; extracting and recording the digital scene information in the real-scene data according to the scene image and text recognition result. The AR terminal collects the screen of the offline scene in real time, and performs accurate scene standard detection and efficient scene image and text recognition on the real-time collected real-scene data, extracting the digital scene information in the real-time scene screen, and realizing efficient and accurate offline scene digital information collection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an offline scene information collection method, platform, system and medium based on an AR terminal. Background Art

[0002] At present, digital construction is becoming increasingly important in urban development. The core of the digital system is to acquire knowledge in information centers through digitalization and data-driven means, so as to create new wealth at a lower cost, higher efficiency and on a larger scale.

[0003] Currently, digital information collection of offline scenes is mainly obtained through manual collection. The most commonly used method is to use a collection APP on a mobile terminal to manually input information, take photos, and shoot videos through manual on-site visits. This method has disadvantages in actual offline operations, such as low efficiency, high error rate of information entry, and difficulty in obtaining complete information. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an offline scene information collection method, platform, system and medium based on AR terminals, aiming to improve the efficiency and accuracy of digital information collection of offline scenes.

[0005] The technical solution of the present invention is as follows:

[0006] A method for collecting offline scene information based on an AR terminal, comprising:

[0007] Receive real scene data collected by the AR terminal;

[0008] Performing scene standard detection on the real scene data to confirm whether the real scene data meets the preset collection standard;

[0009] When the real scene data meets the preset collection standard, performing scene graphic recognition on the real scene data;

[0010] The digitized scene information in the real scene data is extracted and recorded according to the scene image and text recognition result.

[0011] In one embodiment, the method further comprises:

[0012] When the real scene data does not meet the preset collection standard, corresponding collection guidance information is sent to the AR terminal according to the scene standard detection result, so that the AR terminal merges and superimposes the collection guidance information with the real scene data.

[0013] In one embodiment, the collection guide information is at least one of text prompt information, image prompt information and voice prompt information.

[0014] In one embodiment, before performing scene standard detection on the real scene data to confirm whether the real scene data meets the preset acquisition standard, the method further includes:

[0015] The real scene data is subjected to scene prediction according to a preset scene.

[0016] In one embodiment, the performing scene prediction on the real scene data includes:

[0017] Performing scene recognition on the real scene data to obtain a corresponding scene category;

[0018] Determine whether the scene category is a preset scene, and if so, perform scene standard detection.

[0019] In one embodiment, the method further comprises:

[0020] Receiving user interaction information acquired by the AR terminal;

[0021] Identifying the user interaction information to obtain corresponding interaction instructions;

[0022] The real scene data is interactively processed accordingly according to the interactive instruction.

[0023] In one embodiment, the step of extracting and recording the digitized scene information in the real scene data according to the scene image and text recognition result includes:

[0024] Acquire a preset information category in the digital scene information;

[0025] Extracting information from the scene image and text recognition results according to the preset information category to obtain target information in the real scene data;

[0026] The target information is stored in a preset database.

[0027] An offline scene information acquisition platform based on an AR terminal, the platform comprising at least one processor; and

[0028] a memory communicatively connected to the at least one processor; wherein,

[0029] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned offline scene information collection method based on the AR terminal.

[0030] An offline scene information collection system based on an AR terminal, the system comprising a plurality of clients, a server and the offline scene information collection platform as described above; the plurality of clients and the server are all communicatively connected with the offline scene information collection platform.

[0031] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned offline scene information collection method based on AR terminal.

[0032] Beneficial effects: The present invention discloses an offline scene information collection method, platform, system and medium based on an AR terminal. Compared with the prior art, the embodiments of the present invention collect the images of offline scenes in real time through an AR terminal, and perform accurate scene standard detection and efficient scene graphic recognition on the real-time collected real-scene data, extract the digital scene information in the real-time scene image, and realize efficient and accurate offline scene digital information collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0034] Figure 1 A flowchart of an offline scene information collection method based on an AR terminal provided in an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of the hardware structure of an offline scene information collection platform based on an AR terminal provided in an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of the hardware structure of an offline scene information acquisition system based on an AR terminal provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0038] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the offline scene information collection method based on the AR terminal provided by the present invention. Figure 1 As shown, the method specifically comprises the following steps:

[0039] S100: Receive real scene data collected by an AR terminal.

[0040] In this embodiment, the user collects real-time images of offline scenes through AR terminals to obtain corresponding real-scene data. AR (Augmented Reality) technology is a technology that cleverly integrates virtual information with the real world. It widely uses various technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0041] When realizing digital information collection of offline scenes, users can wear AR terminals equipped with cameras such as AR glasses, move their eyes to the scene to be collected according to collection needs, such as shopping malls, stores, street signs, etc., and obtain the real-life data of the scene in real time through the camera on the AR terminal. The data is then transmitted back to the offline scene information collection background for further data processing through data transmission methods such as 5G networks, wifi, Bluetooth, etc., to realize accurate and real-time real-life data collection.

[0042] S200: Perform scene standard detection on the real scene data to confirm whether the real scene data meets the preset collection standard.

[0043] In this embodiment, when the background receives the real-scene data transmitted by the AR terminal, it further processes the data. Taking the real-scene data as a video stream as an example, the AR terminal compresses the video stream acquired in real time and transmits it to the background. The background decodes the received video stream to extract the video picture therein, and further performs preliminary scene standard detection on the video picture to confirm whether the real-scene data collected within the current user's field of view meets the preset collection standard.

[0044] For example, taking the digital information collection of shopping mall stores as an example, information is collected on the storefronts of shopping mall stores. The preset collection standard requires that the real-life data picture must include a complete storefront image, and partial pictures of adjacent stores on the left and right sides need to be leaked to ensure the integrity of the storefront information collection. When the background receives the picture transmitted by the AR terminal, it performs scene standard detection based on the preset collection standard to ensure the correctness and completeness of subsequent scene information collection.

[0045] Of course, the preset collection standard can be flexibly set and adjusted according to different scene collection requirements, and this embodiment does not limit this.

[0046] S300: When the real scene data meets the preset collection standard, perform scene graphic recognition on the real scene data.

[0047] In this embodiment, when the currently collected real-scene data meets the preset collection standard, scene image and text recognition is further performed on the decoded video screen to extract the image and text information therein, so as to realize the digital information collection of offline scenes. Specifically, scene image and text recognition can be realized through a deep learning network. Deep learning (DL) is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to artificial intelligence (AI), that is, through artificial intelligence recognition, the scene images and text recognition of the scene screen collected in real time are efficiently and accurately performed. Taking the information collection of shopping malls and stores as an example, scene recognition, doorhead recognition, LOGO recognition, object recognition, liveness recognition, crowd counting, etc. can be performed. Specifically, the existing deep learning network model can be adopted, and this embodiment does not limit this.

[0048] In one embodiment, the method further comprises:

[0049] When the real scene data does not meet the preset collection standard, corresponding collection guidance information is sent to the AR terminal according to the scene standard detection result, so that the AR terminal merges and superimposes the collection guidance information with the real scene data.

[0050] In this embodiment, if after performing scene standard detection on the received real-scene data, it is determined that the currently collected video picture does not meet the preset collection standard, for example, the store front is missing, the picture is tilted, part of the picture of the adjacent store is not collected, etc., then the background sends corresponding collection guidance information to the AR terminal according to the scene standard detection result. The collection guidance information is at least one of text prompt information, image prompt information and voice prompt information. After receiving the collection guidance information, the AR terminal, based on the function of integrating the virtual picture with the real picture of the AR terminal, superimposes the relevant text prompt information and image prompt information on the real picture, so as to achieve AR picture integration. The user can adjust his or her position through the picture within the current field of view that is integrated with the collection guidance information, that is, synchronously adjust the position of the AR terminal to adjust the real-time collected picture. For example, if it is found after detection that the current store front collection is incomplete and the user is too close to the store front, the collection guidance information of "Please step back a little" can be output, and the corresponding text or direction arrow can be integrated into the current collected real picture. Of course, it can be further combined with voice guidance to allow the user to step back to completely collect the store front picture, until the real scene data meets the preset collection standard, and then enter the next step of scene graphic recognition process to ensure the integrity and accuracy of offline scene information collection.

[0051] S400: extracting and recording the digital scene information in the real scene data according to the scene image and text recognition result.

[0052] In this embodiment, after efficient and accurate scene image and text recognition based on the AI ​​intelligent module, the corresponding digital scene information is extracted and recorded in the recognition result according to the current information collection needs, so that the user only needs to simply wear the AI ​​terminal in the offline scene to adjust the position according to the collection guidance information to complete the digital information collection process of the offline scene, without the need for the user to manually take pictures, fill in information, etc., thereby improving the efficiency and convenience of information collection. After the digital scene information is extracted and recorded, it can be further sent to the AR terminal in this embodiment, and the digital scene information and the real picture, that is, the real scene data, are fused and superimposed in the AR terminal, so that the user can intuitively view the digital scene information within the field of view, providing the user with rich and real-time digital information during the collection process.

[0053] In one embodiment, step S400 includes:

[0054] Acquire a preset information category in the digital scene information;

[0055] Extracting information from the scene image and text recognition results according to the preset information category to obtain target information in the real scene data;

[0056] The target information is stored in a preset database.

[0057] In this embodiment, when extracting digital scene information, preset information categories in the digital scene information are obtained based on the current collection requirements, that is, which digital information needs to be collected at present. Since rich recognition results can be obtained after scene graphic recognition of real-scene data, users can flexibly set corresponding preset information categories according to different collection requirements such as fine collection, rough collection, etc., and then filter and extract the scene graphic recognition results to reduce the amount of data and achieve accurate collection.

[0058] Taking the digital information collection of shopping malls and stores as an example, after scene image and text recognition, six types of recognition results can be obtained, including scene recognition, doorhead recognition, LOGO recognition, object recognition, liveness recognition, and crowd counting. Among them, scene recognition is to determine what scene is in the picture, such as a shopping mall, a restaurant, a company, etc.; doorhead recognition is to determine whether the picture includes a store doorhead; LOGO recognition is to recognize the trademark LOGO in the picture; object recognition is to recognize objects in the picture, such as clothes, cars, etc.; liveness recognition is to recognize whether the picture contains people, animals, etc.; crowd counting is to count the number of people appearing in the picture; based on the collection needs, all categories of information can be extracted, or some categories of information can be extracted, so as to obtain the target information in the real-scene data collected this time, and store the target information in the preset database to realize an efficient and convenient digital information collection process.

[0059] In one embodiment, before step S200, the method further includes:

[0060] The real scene data is subjected to scene prediction according to a preset scene.

[0061] In this embodiment, preset scenes can be set in advance according to different collection requirements. When the user collects real-scene data in real time through the AR terminal, the real-scene data is first predicted based on the preset scene. The prediction process can be performed locally, that is, in the AR terminal, that is, the real-scene data is transmitted to the background after the prediction, or the real-scene data can be directly transmitted to the background, and the scene prediction is performed in the background. This embodiment does not limit this. The preliminary scene prediction is used to determine whether it meets the corresponding scene requirements, so as to save data processing volume and system power consumption as much as possible.

[0062] In one embodiment, performing scene prediction on the real scene data includes:

[0063] Performing scene recognition on the real scene data to obtain a corresponding scene category;

[0064] Determine whether the scene category is a preset scene, and if so, perform scene standard detection.

[0065] In this embodiment, it is preferred to perform scene prediction locally on the AR terminal, obtain real-time video images through the camera, and perform preliminary scene recognition locally through a simple scene recognition algorithm to obtain the scene category within the field of view. Then, it is further determined whether there is content that needs to be collected within the field of view, that is, whether it is a preset scene. If the content within the field of view is a pre-set scene that needs to be collected, the AR terminal starts real-time image collection and sends the collected real-scene data to the background for further scene standard detection and processing. Otherwise, the real-time image will not be transmitted back, so that the user can start the transmission and processing of the real-scene data after arriving at the collection destination, thereby saving system power consumption as much as possible and improving battery life.

[0066] In one embodiment, the method further comprises:

[0067] Receiving user interaction information acquired by the AR terminal;

[0068] Identifying the user interaction information to obtain corresponding interaction instructions;

[0069] The real scene data is interactively processed accordingly according to the interactive instruction.

[0070] In this embodiment, the user may also interact with the AR terminal as needed during the collection process, obtain user interaction information through the AR terminal, and send the obtained user interaction information to the background for further recognition processing, so as to perform corresponding interaction feedback processing after identifying the corresponding interaction instruction. Specifically, the user interaction information may be gesture interaction information, voice interaction information, etc. Since the corresponding interaction, button, prompt information, etc. are all integrated into the user's field of view through a virtual screen when the AR terminal is used to collect information of offline scenes, the user may preferably interact with the AR terminal through gestures. For example, in the process of collecting information at the entrance of a shopping mall store, it is necessary to focus on identifying the local content in the screen. The user may click on the corresponding screen in the field of view through a gesture. The AR terminal obtains the click gesture and the corresponding local screen and sends them to the background. The background recognizes the user's intention and obtains the corresponding interaction instruction, and then further focuses on identifying the local screen, such as enlarging the local screen and processing the clarity, and then identifying the scene image and text therein, etc. Through simple gestures or voice interactions, convenient human-computer interaction can be achieved in the offline information collection process, simplifying human-computer interaction and improving interaction efficiency.

[0071] Another embodiment of the present invention provides an offline scene information collection platform based on an AR terminal, such as Figure 2 As shown, the platform 10 includes:

[0072] One or more processors 110 and memory 120, Figure 2 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0073] The processor 110 is used to complete various control logics of the platform 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0074] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the offline scene information collection method based on the AR terminal in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the platform 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, realizing the offline scene information collection method based on the AR terminal in the above method embodiment.

[0075] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store the application required for the operating platform and at least one function; the data storage area may store data created according to the use of the platform 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the platform 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0076] One or more units are stored in the memory 120, and when executed by one or more processors 110, the offline scene information collection method based on the AR terminal in any of the above method embodiments is executed, for example, the above described method is executed. Figure 1 The method comprises steps S100 to S400.

[0077] Another embodiment of the present invention further provides an offline scene information collection system based on an AR terminal, such as Figure 3 As shown, the system includes the offline scene information acquisition platform 10 and the AR terminal 20 as described above; the AR terminal 20 is communicatively connected to the offline scene information acquisition platform 10, wherein the AR terminal 20 is used to realize the functions of acquiring real-scene data of offline scenes, obtaining user interaction information, pre-judging the real-scene data according to preset scenes, and fusing and superimposing the acquired guidance information with the real-scene data.

[0078] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S100 to S400.

[0079] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0080] In summary, in the offline scene information collection method, platform, system and medium based on AR terminal disclosed in the present invention, the method receives the real scene data collected by the AR terminal; performs scene standard detection on the real scene data to confirm whether the real scene data meets the preset collection standard; when the real scene data meets the preset collection standard, performs scene graphic recognition on the real scene data; extracts and records the digital scene information in the real scene data according to the scene graphic recognition result. The AR terminal collects the screen of the offline scene in real time, and performs accurate scene standard detection and efficient scene graphic recognition on the real scene data collected in real time, extracts the digital scene information in the real-time scene screen, and realizes efficient and accurate offline scene digital information collection.

[0081] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0082] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for collecting offline scene information based on an AR terminal, characterized in that: include: Receive real scene data collected by the AR terminal; Performing scene standard detection on the real scene data to confirm whether the real scene data meets the preset collection standard; When the real scene data meets the preset collection standard, performing scene graphic recognition on the real scene data; The categories of scene image and text recognition results include scene recognition, door head recognition, logo recognition, object recognition, living body recognition and crowd counting; When the real scene data does not meet the preset collection standard, corresponding collection guidance information is sent to the AR terminal according to the scene standard detection result, so that the AR terminal fuses and superimposes the collection guidance information with the real scene data, and adjusts the position of the AR terminal synchronously through the picture within the current field of view and fused with the collection guidance information to adjust the picture collected in real time; Extracting and recording the digital scene information in the real scene data according to the scene image and text recognition result; The step of extracting and recording the digital scene information in the real scene data according to the scene image and text recognition result includes: Acquire a preset information category in the digital scene information; Extracting information from the scene image and text recognition result according to the preset information category, extracting information of all categories in the recognition result, or extracting information of some categories, to obtain target information in the real scene data; The target information is stored in a preset database.

2. The offline scene information collection method based on AR terminal according to claim 1 is characterized in that: The collection guidance information is at least one of text prompt information, image prompt information and voice prompt information.

3. The offline scene information collection method based on AR terminal according to claim 1 is characterized in that: Before performing scene standard detection on the real scene data to confirm whether the real scene data meets the preset collection standard, the method further includes: The real scene data is subjected to scene prediction according to a preset scene.

4. The offline scene information collection method based on AR terminal according to claim 3 is characterized in that: The performing scene prediction on the real scene data includes: Performing scene recognition on the real scene data to obtain a corresponding scene category; Determine whether the scene category is a preset scene, and if so, perform scene standard detection.

5. The offline scene information collection method based on AR terminal according to claim 1 is characterized in that: The method further comprises: Receiving user interaction information acquired by the AR terminal; Identifying the user interaction information to obtain corresponding interaction instructions; The real scene data is interactively processed accordingly according to the interactive instruction.

6. An offline scene information collection platform based on AR terminal, characterized in that: include: The platform includes at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the offline scene information collection method based on the AR terminal as described in any one of claims 1-5.

7. An offline scene information collection system based on AR terminal, characterized in that: The system includes an AR terminal and the offline scene information collection platform as described in claim 6; the AR terminal is communicatively connected to the offline scene information collection platform.

8. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the offline scene information collection method based on the AR terminal as described in any one of claims 1-5.

Images