Bird observation method, system, device and storage medium based on augmented reality

Abstract

The present invention is applicable to the field of augmented reality technology, and provides a bird observation method, system, device and storage medium based on augmented reality. The method includes: obtaining video information of birds; processing the video information according to a computer vision method to obtain identification information of the birds; superimposing the identification information and the video information to obtain a virtual image of the bird; determining the position of the virtual image in the real world according to the user's head movement information, and providing real-time bird identification and rich virtual information superposition through augmented reality-based glasses, so that bird watchers can easily identify birds and understand their characteristics and ecological information, while promoting social interaction and data sharing among bird watchers, and improving the overall bird watching experience.

Description

Technical Field

[0001] The present application belongs to the field of augmented reality technology, and in particular relates to a bird observation method, system, device and storage medium based on augmented reality. Background Art

[0002] Augmented reality technology, also known as AR technology, combines virtual information with the real world to enhance the user's perception of the real environment through computer-generated images, sounds, text or other virtual elements. The goal of AR is to expand the user's perception and interaction, allowing them to interact with the virtual world in real time while retaining their vision of the real world. AR technology has made significant progress in various fields, including entertainment, healthcare, education, etc. With the improvement of hardware and the development of algorithms, AR technology has become more and more mature and popular.

[0003] The application of existing augmented reality technology in the field of bird observation is generally to better introduce birds through augmented reality after the bird species are known. It is generally used in bird museums or zoos and other places. Traditional bird observation in the wild usually relies on telescopes and bird atlases, which requires observers to have high bird identification skills.

[0004] It can be seen that in the prior art, it is difficult to identify birds in the wild, information acquisition and recording are cumbersome and inconvenient, and birds cannot be identified quickly. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a bird observation method based on augmented reality, aiming to solve the problems of difficulty in identifying birds in the wild, cumbersome and inconvenient information acquisition and recording, and inability to quickly identify birds.

[0006] The embodiment of the present application is implemented as follows: a bird observation method based on augmented reality, the method comprising:

[0007] Get video information about birds;

[0008] Processing the video information according to a computer vision method to obtain identification information of the bird;

[0009] superimposing the identification information and the video information to obtain a virtual image of the bird;

[0010] The position of the virtual image in the real world is determined according to the user's head movement information.

[0011] Another object of the embodiment of the present application is that the system comprises:

[0012] Camera, used to obtain video information of birds;

[0013] A core processing module is used to process the video information according to a computer vision method to obtain identification information of the bird; and to superimpose the identification information and the video information to obtain a virtual image of the bird;

[0014] The display module is used to determine the position of the virtual image in the real world according to the user's head movement information.

[0015] Another object of an embodiment of the present application is a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the bird observation method based on augmented reality.

[0016] Another object of an embodiment of the present application is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the augmented reality-based bird observation method.

[0017] The augmented reality-based bird observation method provided in the embodiment of the present application provides real-time bird identification and rich virtual information overlay through AR glasses, so that bird watchers can easily identify birds without flipping through reference materials, thereby greatly improving the efficiency of bird watching; the overlay of virtual information enriches the bird watching experience, and users can instantly obtain detailed information about birds, making bird watching more interactive and educational; users can easily record observed birds and share observation data, thereby promoting social interaction, data sharing, and bird protection work; AR glasses are easy to carry and support instant uploading and sharing of observation data, making outdoor bird watching more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a bird observation method based on augmented reality provided in an embodiment of the present application;

[0019] Figure 2 A video information processing flow chart of a bird observation method based on augmented reality provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of a virtual image presentation of a bird observation method based on augmented reality provided in an embodiment of the present application;

[0021] Figure 4 A structural block diagram of a bird observation system based on augmented reality provided in an embodiment of the present application;

[0022] Figure 5 A flowchart of a network module processing of a bird observation system based on augmented reality provided in an embodiment of the present application;

[0023] Figure 6 A control mode diagram of a control module of a bird observation system based on augmented reality provided in an embodiment of the present application;

[0024] Figure 7 A data storage and processing flow chart of a bird observation method based on augmented reality provided in an embodiment of the present application;

[0025] Figure 8 An AR glasses device for a bird observation system based on augmented reality provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of bird observation based on augmented reality provided in an embodiment of the present application;

[0027] Fig.10 FIG. 4 is a block diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0030] like Figure 1 As shown, in one embodiment, a bird observation method based on augmented reality is proposed, which may specifically include the following steps:

[0031] Step S102, obtaining video information of birds.

[0032] In this embodiment, the video information of birds is captured by a picture taking device such as a camera provided on the smart glasses to capture the bird scene on site and send it to the video processing module. The delay in image capture and information conversion is less than 5ms.

[0033] Step S104: Process the video information according to a computer vision method to obtain bird identification information.

[0034] In this embodiment, the computer vision method is a method for identifying and processing video information, using the opencv framework to process the video information, and using SIFT, SURF, ORB or FAST algorithm to extract features and track the video information.

[0035] Step S106, superimposing the identification information and the video information to obtain a virtual image of the bird.

[0036] In this embodiment, a head-mounted display or projection technology is used to overlay virtual information directly into the user's field of view, including text, images, icons or 3D models to display the name, characteristics and ecological information of the bird.

[0037] Step S108, determining the position of the virtual image in the real world according to the user's head movement information.

[0038] In this embodiment, the user's head movement information is key information that reflects the user's head turning and field of view changes, and is the key to achieving stable superposition of the virtual image in the real world, allowing the user to interact with the virtual content without losing the perception of the real world.

[0039] In one embodiment, Figure 2 As shown, step S104 may specifically include the following steps:

[0040] Step S202, obtaining a video frame sequence according to the video information;

[0041] Step S204, preprocessing the video frame sequence to obtain a preprocessed image;

[0042] Step S206, target detection of the pre-processed image to obtain the bird position;

[0043] Step S208, tracking the position of the bird to obtain the bird's movement trajectory;

[0044] Step S210, feature extraction of the bird's motion trajectory to obtain bird image features;

[0045] Step S212: identifying the features of the bird image according to a preset convolutional neural network to obtain identification information of the bird.

[0046] In this embodiment, the original video frame sequence is obtained from the video information, and inter-frame compression and intra-frame compression are performed. When a frame image is compressed intra-frame, only the data of the current frame is considered without considering the redundant information between adjacent frames, which is similar to static image compression. A lossy compression algorithm is used. The inter-frame compression is achieved based on the characteristic that adjacent frames of a continuous video have redundant information. The compression is performed by comparing the data between different frames on the time axis to further improve the compression ratio. Each frame image is pre-processed, including resizing, contrast enhancement, etc., to improve the image quality and facilitate subsequent processing; Haar target detection in opencv is used to detect bird targets for each pre-processed frame to obtain the bird position; and an optical flow tracking algorithm is used to track the detected birds to obtain their motion trajectories; feature extraction is performed on the birds in the process of the bird motion trajectory, and the features of the birds include: color, texture, shape, etc., and a Distinguished feature vector is extracted; the extracted bird features are input into a pre-trained bird recognition CNN model, and the model analyzes and classifies the feature vectors and outputs the recognition information of the bird, and the recognition information includes: including the name of the bird, drawing an object frame, displaying the motion trajectory, etc.

[0047] In one embodiment, Figure 3 As shown, determining the position of the virtual image in the real world according to the user's head movement information includes:

[0048] Step S302, determining the position where the virtual image is displayed according to the user's head sensor data;

[0049] Step S304: ensuring that the virtual image moves along with the user's head movement according to the user's head movement information.

[0050] In this embodiment, the virtual image of the bird is generated by combining identification information and video information, wherein the identification information includes: bird category, feature information and ecological information; the video information includes: coordinate position, frame information and time information. The virtual image is directly superimposed on the user's field of view using a head-mounted display or projection technology, including text, images, icons or 3D models to display the name, features and ecological information of the bird, and the user's position, direction and field of view angle are identified in real time using sensor data such as gyroscopes and accelerometers, and the position and scale are calibrated to ensure that the virtual image is accurately aligned with the user's field of view, regardless of whether the user moves; based on the continuous monitoring of the user's head movement information, the position and direction of the virtual image are adjusted in real time to maintain the stability of the virtual image in the field of view, and move with the user's head movement. The virtual image can be transparently or semi-transparently superimposed on the user's real world, usually through the optical system of AR glasses, so that the user can see the virtual information and the surrounding environment at the same time.

[0051] In one embodiment, the augmented reality-based bird observation method further includes: calling a remote upload interface to upload the video information and the identification information; and establishing a structured database of the video information and the identification information.

[0052] In this embodiment, a button can be controlled to initiate a network backhaul process, the remote upload interface can be called to upload video information, and the remote data submission interface can be called to submit the current identification information, which includes: frame information, identification data, auxiliary information, etc. The uploaded video information is stored as OSS object storage, and a structured database is established for the uploaded identification information. Data can be called from each other between the OSS object storage and the structured database.

[0053] like Figure 4 As shown, a bird observation system based on augmented reality is provided, and the bird observation system based on augmented reality includes:

[0054] Camera 100, used to obtain video information of birds;

[0055] The core processing module 200 is used to process the video information according to a computer vision method to obtain identification information of the bird; and to superimpose the identification information and the video information to obtain a virtual image of the bird;

[0056] The display module 300 is used to determine the position of the virtual image in the real world according to the user's head movement information.

[0057] In this embodiment, the camera captures video information and transmits it to the core processing module; the core processing module contains a gyroscope, an accelerometer, an NPU, and a nanoSSD. The gyroscope is used to detect the movement of the user's head or eyes, and track the user's head turning and field of view changes, which is the key to achieving stable superposition of the virtual image, allowing the user to interact with the virtual content without losing the perception of the real world; the accelerometer helps the system detect the user's linear acceleration, such as forward and backward, left and right, or up and down movement, for the position tracking and interaction of the virtual image; the NPU is used to process image recognition, object tracking, bird recognition and other complex computing tasks in real time, and analyze the image data captured by the camera. The nanoSSD uses solid-state drive (SSD) technology to store applications, operating systems, user data, and virtual images.

[0058] In this embodiment, the display module presents the virtual image through a transparent display surface, allowing light to be transmitted through the virtual image to maintain the perception of the real world. The virtual image is projected into the user's field of view using projection technology to ensure that the virtual image is aligned with the user's field of view. The data of the gyroscope and accelerometer are used to track the user's head movement in real time and adjust the position and direction of the virtual information accordingly. The virtual image is superimposed on the user's field of view in the form of images and texts to ensure that the virtual image interacts with objects in the real world.

[0059] In this embodiment, the system further includes:

[0060] The control module 400 is used to control the camera, the core processing module and the display module.

[0061] The network module 500 is used to call a remote upload interface to upload the video information and the identification information.

[0062] In this embodiment, click the control button to call the relevant functions of the control module. Click once on the observation interface to upload the video, ensuring the speed of uploading. Click twice in succession to pop up the menu interface, which has three functions: 1. View records; 2. View interactive information; 3. Turn on and off. Click the button to select the function to enter and view the relevant interface.

[0063] An augmented reality-based bird observation method and system provided in an embodiment of the present application integrates image recognition, graphic rendering, and final drawing and overlaying of images in augmented reality technology on AR smart glasses, and overlays bird scenes and their virtual images on AR smart glasses, which can identify birds in real time and provide identification results, lowering the threshold for bird identification and solving the problem of difficulty in bird identification; providing instant and customizable bird information, so that users can easily obtain detailed information about birds, solving the problem of inconvenient information acquisition; automatically recording observed birds, including taking photos and videos, thereby simplifying the data recording process, solving the problem of cumbersome data recording, and easily sharing data through smart phones or other devices, and promoting social interaction and data sharing among bird watchers, solving the problem of social and sharing barriers.

[0064] Fig.10 FIG. 2 shows an internal structure diagram of a computer device in one embodiment. Fig.10As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement a bird observation method based on augmented reality. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can execute a bird observation method based on augmented reality. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0065] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0066] In one embodiment, the bird observation system based on augmented reality provided by the present application can be implemented in the form of a computer program. The computer program can be used in Fig.10 The computer device shown in the figure is run on the computer device. The memory of the computer device can store various program modules constituting the bird observation system based on augmented reality, such as: Figure 4 The camera, core processing module and display module shown in the figure. The computer program composed of various program modules enables the processor to execute the steps of the bird observation method based on augmented reality in various embodiments of the present application described in this specification.

[0067] In one embodiment, a computer device is provided, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0068] Get video information about birds;

[0069] Processing the video information according to a computer vision method to obtain identification information of the bird;

[0070] superimposing the identification information and the video information to obtain a virtual image of the bird;

[0071] The position of the virtual image in the real world is determined according to the user's head movement information.

[0072] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor performs the following steps:

[0073] Get video information about birds;

[0074] Processing the video information according to a computer vision method to obtain identification information of the bird;

[0075] superimposing the identification information and the video information to obtain a virtual image of the bird;

[0076] The position of the virtual image in the real world is determined according to the user's head movement information.

[0077] It should be understood that, although each step in the flow chart of each embodiment of the present application is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0078] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0079] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bird observation method based on augmented reality, characterized in that: The method comprises: Get video information about birds; Processing the video information according to a computer vision method to obtain identification information of the bird; superimposing the identification information and the video information to obtain a virtual image of the bird; Determining the position of the virtual image in the real world according to the user's head movement information; The identification information includes: bird category, characteristic information and ecological information; the video information includes: coordinate position, frame information and time information; The step of determining the position of the virtual image in the real world according to the user's head movement information includes: Determining a position where the virtual image is displayed based on user head sensor data; According to the user's head movement information, ensuring that the virtual image moves along with the user's head movement; The step of processing the video information according to a computer vision method to obtain bird identification information comprises the following steps: Obtaining a video frame sequence according to the video information; Preprocessing the video frame sequence to obtain a preprocessed image, wherein the preprocessing includes: inter-frame compression and frame compression; Target detection of the preprocessed image to obtain the location of the bird; Tracking the position of the bird to obtain the bird's movement trajectory; Feature extraction of the bird's motion trajectory to obtain bird image features; According to a preset convolutional neural network, the features of the bird image are identified to obtain identification information of the bird; The original video frame sequence is obtained from the video information. Through inter-frame compression and intra-frame compression, when a frame of image is compressed intra-frame, only the data of the current frame is considered without considering the redundant information between adjacent frames. A lossy compression algorithm is used. Inter-frame compression is based on the characteristic that there is redundant information between adjacent frames of continuous video. Compression is implemented by comparing the data between different frames on the time axis to further improve the compression ratio. Each frame of image is pre-processed; Use the optical flow tracking algorithm to track the detected birds and obtain their motion trajectories; extract the features of the birds in the bird motion trajectory and extract the feature vector; input the extracted bird features into the pre-trained bird recognition CNN model, the model analyzes and classifies the feature vectors and outputs the bird recognition information.

2. The bird observation method based on augmented reality according to claim 1, characterized in that: The method further comprises: Call a remote upload interface to upload the video information and the identification information.

3. The bird observation method based on augmented reality according to claim 1, characterized in that: The method further comprises: A structured database of the video information and the identification information is established.

4. A bird observation system based on augmented reality, characterized in that: The system comprises: Camera, used to obtain video information of birds; A core processing module, used for processing the video information according to a computer vision method to obtain identification information of the bird; and superimposing the identification information and the video information to obtain a virtual image of the bird; A display module, used to determine the position of the virtual image in the real world according to the user's head movement information; Core processing module, specifically used for: Obtaining a video frame sequence according to the video information; Preprocessing the video frame sequence to obtain a preprocessed image, wherein the preprocessing includes: inter-frame compression and frame compression; Target detection of the preprocessed image to obtain the location of the bird; Tracking the position of the bird to obtain the bird's movement trajectory; Feature extraction of the bird's motion trajectory to obtain bird image features; According to a preset convolutional neural network, the features of the bird image are identified to obtain identification information of the bird; The original video frame sequence is obtained from the video information. Through inter-frame compression and intra-frame compression, when a frame of image is compressed intra-frame, only the data of the current frame is considered without considering the redundant information between adjacent frames. A lossy compression algorithm is used. Inter-frame compression is based on the characteristic that there is redundant information between adjacent frames of continuous video. Compression is implemented by comparing the data between different frames on the time axis to further improve the compression ratio. Each frame of image is pre-processed; Use the optical flow tracking algorithm to track the detected birds and obtain their motion trajectories; extract the features of the birds in the bird motion trajectory and extract the feature vector; input the extracted bird features into the pre-trained bird recognition CNN model, the model analyzes and classifies the feature vectors and outputs the bird recognition information.

5. The bird observation system based on augmented reality according to claim 4, characterized in that: The system further comprises: A control module, used to control the camera, the core processing module and the display module; The network module is used to call the remote upload interface to upload the video information and the identification information.

6. A computer device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the bird observation method based on augmented reality as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the bird observation method based on augmented reality as described in any one of claims 1 to 3.

Images