Intelligent agent autonomous tracing and homeward voyage method and device, electronic equipment and storage medium

Abstract

The invention relates to an agent autonomous tracking and homeward voyage scheme, relates to the technical field of artificial intelligence, and solves the technical problems that an existing unmanned aerial vehicle autonomous homeward voyage technology is greatly influenced by accumulated drifting, and the computing power demand is large. The method comprises the following steps in an agent return phase: step S101, reading a forward environment image; step S102, extracting feature points of the outbound environment image, and matching the feature points with the last element read from the reference image feature point queue; s103, calculating the rotation amount and the translation amount of the intelligent agent according to a matching result; s104, judging whether the rotation amount and the translation amount of the intelligent agent meet threshold requirements or not, if yes, generating an intelligent agent rotation instruction or translation instruction according to the rotation amount and the translation amount, removing the last element from the reference image feature point queue and returning to the step S101; and otherwise, directly referring to the last element removed from the image feature point queue, and returning to the step S101.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for autonomous tracking and returning of an intelligent entity. Background Art

[0002] In many cases, drones equipped with Global Navigation Satellite System (GNSS) positioning function may suddenly face the situation that GNSS cannot be used after flying for a long time. In the civilian field, when drones fly into dense forests and other environments where GNSS cannot be used, GNSS positioning will fail because 4 or more satellites cannot be seen. In this case, how to make the drone return home autonomously and land successfully is a technical problem that needs to be solved urgently.

[0003] Existing methods generally adopt technical strategies based on visual inertial odometry, which mainly utilizes the data collected by the camera mounted under the drone and the inertial gyroscope sensor on the aircraft to generate the drone's outbound route. When the GNSS signal is lost, the aircraft follows the outbound route to achieve autonomous return function.

[0004] Since existing methods rely on the visual inertial odometry (VIO) algorithm and the back-end optimization steps in the simultaneous localization and mapping (SLAM) algorithm, the algorithm complexity is high, so the computing power requirements are large. In addition, since existing methods require inertial gyroscope sensor data, and inertial gyroscope sensor data is often affected by cumulative drift, existing methods are greatly affected by cumulative drift. Summary of the invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a method, device, electronic device and storage medium for autonomous tracking and returning of an intelligent entity, so as to solve the technical problems that the existing autonomous return technology of unmanned aerial vehicles is greatly affected by cumulative drift and requires high computing power.

[0006] In a first aspect, an embodiment of the present invention provides a method for autonomous tracking and returning of an intelligent agent, wherein the following steps are performed in the outbound stage of the intelligent agent:

[0007] Step S101: reading the return environment image acquired by the image acquisition device;

[0008] Step S102: extracting feature points of the return environment image, and matching the feature points of the return environment image with the last element read from the reference image feature point queue, wherein the reference image feature point queue sequentially stores reference image feature points of the agent at multiple moments in the outbound phase;

[0009] Step S103: Calculate the rotation and translation of the intelligent body according to the matching results;

[0010] Step S104: Determine whether the rotation amount of the intelligent body exceeds the first rotation threshold and whether the translation amount of the intelligent body exceeds the first translation threshold. If the rotation amount of the intelligent body exceeds the first rotation threshold or the translation amount of the intelligent body exceeds the first translation threshold, generate an intelligent body rotation instruction or a translation instruction according to the rotation amount or the translation amount, remove the last element from the reference image feature point queue and return to step S101; otherwise, directly remove the last element from the reference image feature point queue and return to step S101.

[0011] Based on the further improvement of the above method, in the agent outbound stage, the reference image feature point queue is obtained by the following method:

[0012] Step S100: reading the outbound environment image collected by the image acquisition device;

[0013] Step S200: extracting feature points of the outbound environment image, and matching the feature points of the outbound environment image with the last element read from the reference image feature point queue; wherein the reference image feature point queue stores feature points of one or more reference images, and the first element of the reference image feature point queue is the feature point of the first outbound environment image;

[0014] Step S300: Calculate the rotation and translation of the intelligent body according to the matching results;

[0015] Step S400: Determine whether the rotation amount of the intelligent body exceeds the second rotation threshold or whether the translation amount of the intelligent body exceeds the second translation threshold. If the rotation amount of the intelligent body exceeds the second rotation threshold or the translation amount of the intelligent body exceeds the second translation threshold, set the outbound environment image as a reference image and store the feature points of the outbound environment image into the reference image feature point queue, wherein the reference image includes the satellite navigation position information of the outbound environment image; otherwise, enter step S100.

[0016] Based on the further improvement of the above method, in the outbound stage or the return stage of the intelligent agent, the extracting feature points of the outbound environment image or the return environment image includes:

[0017] The SuperPoint feature detection algorithm is used to extract feature points of the outbound environment image or the return environment image.

[0018] Based on the further improvement of the above method, in the outbound stage or the return stage of the intelligent agent, matching the feature points of the outbound environment image or the return environment image with the last element read from the reference image feature point queue includes:

[0019] The SuperGlue image matching algorithm is used to match the feature points of the outbound environment image or the return environment image with the last element read from the reference image feature point queue.

[0020] Based on the further improvement of the above method, in the outbound stage or the return stage of the intelligent agent, before extracting the feature points of the outbound environment image or the return environment image, the following steps are also included:

[0021] Determine whether the time interval between the outbound environment image or the return environment image and the outbound environment image or the return environment image at the previous moment meets the interval threshold; if so, enter the step of extracting feature points of the outbound environment image or the return environment image; otherwise, read the outbound environment image or the return environment image captured by the image capture device at the next moment.

[0022] Based on a further improvement of the above method, the interval threshold is determined by the time interval between each element in the reference image feature point queue.

[0023] Based on the further improvement of the above method, in the outbound stage or the return stage of the intelligent agent, the extracting feature points of the outbound environment image or the return environment image includes:

[0024] Extract feature points of a central rectangular area of ​​the outbound environmental image or the return environmental image.

[0025] Based on the further improvement of the above method, the intelligent agent is a drone, and in the outbound stage or the return stage of the drone, before extracting the feature points of the outbound environment image or the return environment image, the following steps are also included:

[0026] Mask processing is performed on the landing gear of the UAV in the outbound environmental image or the return environmental image.

[0027] In a second aspect, an embodiment of the present invention provides an intelligent body autonomous tracking and returning device, the device comprising:

[0028] A first reading module is configured to read the return environment image acquired by the image acquisition device;

[0029] A first extraction and matching module is configured to extract feature points of the return environment image and match the feature points of the return environment image with the last element read from a reference image feature point queue, wherein the reference image feature point queue sequentially stores reference image feature points of the agent at multiple moments in the outbound phase;

[0030] A first calculation module is configured to calculate the rotation amount and / or translation amount of the intelligent body according to the matching result;

[0031] The judgment and processing module is configured to judge whether the rotation amount of the intelligent body exceeds the first rotation threshold and whether the translation amount of the intelligent body exceeds the first translation threshold. If the rotation amount of the intelligent body exceeds the first rotation threshold or the translation amount of the intelligent body exceeds the first translation threshold, an intelligent body rotation instruction or a translation instruction is generated according to the rotation amount or the translation amount, and the last element is removed from the reference image feature point queue and the first reading module is called; otherwise, the last element is directly removed from the reference image feature point queue and the first reading module is called.

[0032] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0033] at least one processor; and

[0034] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent body autonomous tracking and returning method as described in any one of the first aspect of the present invention.

[0035] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the intelligent body autonomous tracking and returning method as described in any one of the first aspect of the present invention.

[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0037] 1. The solution of the present invention only requires an image acquisition device for taking images, and does not use the visual inertial odometry (VIO) algorithm and the back-end optimization steps in the simultaneous localization and mapping (SLAM) algorithm. Therefore, the algorithm complexity is lower than that of the visual inertial odometry calculation method, and the computing power requirement is relatively small. It has great application value in small intelligent entities (such as small drones) with limited computing power resources.

[0038] 2. The solution of the present invention does not use inertial gyroscope sensor data, so it is less affected by the cumulative drift, thereby improving the success rate of the intelligent body's autonomous return.

[0039] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0041] Figure 1a and Figure 1b A schematic diagram of the process of an intelligent agent autonomous tracking and returning method according to an embodiment of the present invention is shown.

[0042] Figure 2a and Figure 2b A schematic flow chart of an intelligent agent autonomous tracking and returning method according to another embodiment of the present invention is shown.

[0043] Figure 3 An exemplary block diagram of an intelligent agent autonomous tracking and returning device according to an embodiment of the present invention is shown.

[0044] Figure 4 A schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0046] Figure 1a A schematic diagram of the process of an intelligent agent autonomous tracking and returning method according to an embodiment of the present invention is shown. Figure 1a The figure shows a flow chart of the method for autonomous tracking and returning of an intelligent body according to an embodiment of the present invention during the intelligent body's return stage.

[0047] like Figure 1a As shown, in the intelligent agent return stage, the intelligent agent autonomous tracking and returning method includes the following steps:

[0048] Step S101: reading the return environment image captured by the image acquisition device.

[0049] In this embodiment, the image acquisition device may be a camera sensor. In this embodiment, the agent may acquire images at discrete moments of autonomous selection. It should be noted that during the return phase, the agent does not have a GNSS signal, so in this embodiment, the return environment image read does not have location information.

[0050] In some embodiments, the intelligent agent may be a drone, and the image acquisition device may be a camera sensor that shoots downward. In some embodiments, the timing for reading the return environment image may be when the overlap rate of two return environment images continuously acquired by the image acquisition device is greater than a preset threshold (e.g., 10%).

[0051] Step S102: extracting feature points of the return environment image, and matching the feature points of the return environment image with the last element read from the reference image feature point queue.

[0052] In this embodiment, a feature point extraction algorithm based on a deep neural network can be used to extract feature points of the return environment image. In some embodiments, a SuperPoint network is preferably used to extract feature points of the return environment image. Specifically, the return environment image can be input into the SuperPoint network, and the output of the last layer structure of the SuperPoint network is used as a feature point.

[0053] Below are T1, T2, T3, ..., T M Step S102 is described by taking M return environment images read at consecutive moments as an example.

[0054] First, the feature points of the first return environment image read at time T1 can be extracted, and then the feature points of the first return environment image are matched with the last element read from the reference image feature point queue. In this embodiment, a deep learning model for image feature point matching can be used to match the feature points between the two images. In some embodiments, a SuperGlue deep learning model can be used to match feature points.

[0055] The following is a description of a method for generating a reference image feature point queue in an embodiment of the present invention.

[0056] Figure 1b FIG. 2 is a flow chart showing a method for generating a reference image feature point queue according to an embodiment of the present invention. Figure 1b As shown in the figure, in the agent outbound phase, the following steps are performed:

[0057] Step S100: reading the outbound environment image captured by the image acquisition device.

[0058] In this embodiment, the image acquisition device may be a camera sensor. In this embodiment, the agent may acquire images at discrete moments of autonomous selection. It should be noted that, in the outbound phase, the agent has a GNSS signal, so in this embodiment, the location information of the agent when acquiring the outbound environment image may be read while reading the outbound environment image acquired by the image acquisition device. The location information here may include longitude, latitude, and altitude data.

[0059] In some embodiments, the intelligent agent may be a drone, and the image acquisition device may be a camera sensor that shoots downward. In some embodiments, the timing for reading the outbound environment image may be when the overlap rate of two outbound environment images continuously acquired by the image acquisition device is greater than a preset threshold (e.g., 10%).

[0060] Step S200: extracting feature points of the outbound environment image, and matching the feature points of the outbound environment image with the last element read from the reference image feature point queue.

[0061] In this embodiment, a feature point extraction algorithm based on a deep neural network can be used to extract feature points of the outbound environment image. In some embodiments, a SuperPoint network can be used to extract feature points of the outbound environment image. Specifically, the outbound environment image can be input into the SuperPoint network, and the output of the last layer of the SuperPoint network structure is used as a feature point. The SuperPoint network is a deep learning network for feature point detection and description, which includes a shared encoder and two decoders, which are used for interest point detection and descriptor generation, respectively. The SuperPoint network is able to run on full-size images and simultaneously calculate pixel-level interest point locations and associated descriptors in a forward pass.

[0062] In this embodiment, a stack can be configured to store the feature points of the reference image. N The contents stored in the stack in this embodiment are described by taking N outbound environment images read at consecutive moments as an example.

[0063] First, the feature points of the first outbound environment image read at time t1 can be extracted, and then the first outbound environment image is set as the reference image and the feature points of the first outbound environment image are stored in the stack. The feature points of the first outbound environment image are the first element in the stack.

[0064] After reading the second outbound environment image at time t2, the feature points of the second outbound environment image can be extracted and matched with the latest elements in the stack (i.e., the feature points of the first outbound environment image). In this embodiment, a deep learning model for image feature point matching can be used to match the feature points between the two outbound environment images. In some embodiments, the SuperGlue deep learning model can be used to match feature points. The SuperGlue network is a deep learning model for image feature point matching, which learns feature point matching through graph neural networks (GNNs), and can effectively handle feature point correspondence problems in complex scenes. The SuperGlue model can output the matching relationship between the feature points of the two outbound environment images.

[0065] Next, it can be determined whether to set the second outbound environment image as a reference image (described below) based on the matching result of the feature points of the second outbound environment image and the feature points of the first outbound environment image. If the matching result is to set the second outbound environment image as a reference image, the feature points of the second outbound environment image can be stored in the stack as the second element in the stack.

[0066] After reading the third outbound environment image at time t3, the feature points of the third outbound environment image can be extracted and matched with the latest element in the stack (i.e., the feature points of the second outbound environment image). Next, based on the matching result of the feature points of the third outbound environment image and the feature points of the second outbound environment image, it can be determined whether to set the third outbound environment image as a reference image (to be described below). If the matching result is to set the third outbound environment image as a reference image, the feature points of the third outbound environment image can be stored in the stack as the third element in the stack. After t3, …, t N The stack in this embodiment is a reference image feature point queue, which is used to store feature points of one or more reference images. The first element of the reference image feature point queue is the feature point of the first outbound environment image, and a first-in-last-out mode is adopted.

[0067] Step S300: Calculate the rotation and translation of the agent according to the matching results.

[0068] In step S200, the matching feature point pairs of the two outbound environment images can be obtained. In this embodiment, the displacement vectors of each matching feature point pair can be calculated respectively, and then the average feature point displacement vector of the matching feature point pairs of the two outbound environment images can be obtained by taking the average value, and finally the Euclidean distance of the two outbound environment images is calculated based on the obtained average feature point displacement vector. The translation amount in this embodiment is the Euclidean distance of the two outbound environment images.

[0069] In this embodiment, the three-dimensional rotation matrix of the two outbound environment images can be calculated based on the matching feature point pairs of the two outbound environment images, and then the rotation values ​​of the two outbound environment images in the three-dimensional space are calculated based on the obtained three-dimensional rotation matrix. The rotation amount in this embodiment is the rotation value of the two outbound environment images in the three-dimensional space.

[0070] In some embodiments, considering that most of the drone return scenes have a certain height, and the different heights of surface objects have little effect on the drone, when calculating the rotation amount of the drone, only the rotation amount of the drone in the two-dimensional plane (i.e., the longitude and latitude plane) can be calculated. In this embodiment, the two-dimensional rotation of the plane part (i.e., the longitude and latitude plane) can be selected in the three-dimensional rotation matrix, and the rotation angle value can be calculated. The rotation angle value in the two-dimensional plane is the rotation amount in this embodiment. By reducing the rotation amount of the drone from three dimensions to two dimensions, the algorithm complexity and computing power requirements can be further reduced while ensuring accuracy.

[0071] Step S400: Determine whether the rotation amount of the intelligent body exceeds the second rotation threshold or whether the translation amount of the intelligent body exceeds the second translation threshold. If the rotation amount of the intelligent body exceeds the second rotation threshold or the translation amount of the intelligent body exceeds the second translation threshold, set the outbound environment image as a reference image and store the feature points of the outbound environment image into the reference image feature point queue, wherein the reference image includes the satellite navigation position information of the outbound environment image; otherwise, enter step S100.

[0072] The second rotation threshold can be adaptively linearly determined according to the height of the intelligent body from the ground, and the second translation threshold can be adaptively linearly determined according to the height of the intelligent body from the ground. In some embodiments, at the same height from the ground, the second rotation threshold is smaller than the first rotation threshold to be described below, and the second translation threshold is smaller than the first translation threshold to be described below. In this embodiment, by setting the return threshold more strictly, it can be ensured that the aircraft will not be unable to return due to a wrong match.

[0073] In this embodiment, the rotation amount of the intelligent body obtained in step S300 can be compared with the second rotation threshold. If the rotation amount is less than the second rotation threshold, the process can return to step S100 and start reading the outbound environment image at the next moment; otherwise, the outbound environment image at the current moment can be set as the latest reference image and stored in the reference image feature point queue.

[0074] In this embodiment, the translation amount of the intelligent agent obtained in step S300 can be compared with the second translation threshold. If the translation amount is less than the second translation threshold, the process can return to step S100 and start reading the outbound environment image at the next moment; otherwise, the outbound environment image at the current moment can be set as the latest reference image and stored in the reference image feature point queue.

[0075] In this embodiment, each reference image may include satellite navigation position information corresponding to the outbound environment image. The satellite navigation position information of each reference image may be acquired through a GNSS signal.

[0076] Returning to step S102, next, based on the matching result between the feature points of the first return environment image and the last element in the reference image feature point queue, it can be determined whether to generate a rotation instruction or a translation instruction (described below) for controlling the intelligent agent to adjust its position. If the matching result is to generate a rotation instruction or a translation instruction, then after the intelligent agent adjusts its position according to the rotation instruction or the translation instruction, it can read the second return environment image at time T2 and after extracting the feature points of the second return environment image, continue to match the feature points of the second return environment image with the last element in the reference image feature point queue until the matching result between the feature points of the i-th (i∈[2,M]) return environment image and the last element in the reference image feature point queue is: no rotation instruction or translation instruction needs to be generated. At this point, the intelligent agent has reached the position marked by the last element in the reference image feature point queue. At this time, the last element in the reference image feature point queue can be discarded and the reference image feature point queue can be updated. Afterwards, T can be read. i+1 The i+1th return environment image at the moment, and after extracting the feature points of the i+1th return environment image, continue to match the feature points of the i+1th return environment image with the last element in the updated reference image feature point queue, and repeat the above process until the intelligent agent reaches the position marked by the first element in the reference image feature point queue (i.e., the feature points of the first return environment image read at the moment t1 in step S200), thereby realizing return.

[0077] Step S103: Calculate the rotation and translation of the agent according to the matching results.

[0078] In step S102, the matching feature point pairs of the return environment image and the reference image can be obtained. In this embodiment, the displacement vectors of each matching feature point pair can be calculated respectively, and then the average feature point displacement vector of the matching feature point pairs of the two images can be obtained by taking the average value, and finally the Euclidean distance of the two images is calculated based on the obtained average feature point displacement vector. The translation amount in this embodiment is the Euclidean distance of the two images.

[0079] In this embodiment, the three-dimensional rotation matrix of the two images can be calculated based on the matching feature point pairs of the return environment image and the reference image, and then the rotation value of the two images in the three-dimensional space is calculated based on the obtained three-dimensional rotation matrix. The rotation amount in this embodiment is the rotation value of the two images in the three-dimensional space.

[0080] In some embodiments, considering that most of the drone return scenes have a certain height, and the different heights of surface objects have little effect on the drone, when calculating the rotation amount of the drone, only the rotation amount of the drone in the two-dimensional plane (i.e., the longitude and latitude plane) can be calculated. In this embodiment, the two-dimensional rotation of the plane part (i.e., the longitude and latitude plane) can be selected in the three-dimensional rotation matrix, and the rotation angle value can be calculated. The rotation angle value in the two-dimensional plane is the rotation amount in this embodiment. By reducing the rotation amount of the drone from three dimensions to two dimensions, the algorithm complexity and computing power requirements can be further reduced while ensuring accuracy.

[0081] Step S104: In response to the rotation amount of the intelligent agent exceeding the first rotation threshold or the translation amount of the intelligent agent exceeding the first translation threshold, generate an intelligent agent rotation instruction or a translation instruction according to the rotation amount or the translation amount, remove the last element from the reference image feature point queue and return to step S101; otherwise, directly remove the last element from the reference image feature point queue and return to step S101.

[0082] The first rotation threshold can be adaptively linearly determined according to the height of the intelligent body from the ground, and the first translation threshold can be adaptively linearly determined according to the height of the intelligent body from the ground.

[0083] In this embodiment, the rotation amount of the agent obtained in step S103 can be compared with the second rotation threshold. If the rotation amount is greater than the first rotation threshold, the rotation amount can be returned to the agent for position adjustment. After the agent adjusts its position, it can return to step S101 and start reading the return environment image at the next moment until the result that the rotation amount is equal to or less than the first rotation threshold is obtained. In this embodiment, the translation amount of the agent obtained in step S103 can be compared with the first translation threshold. If the rotation amount is greater than the first translation threshold, the translation amount can be returned to the agent for position adjustment. After the agent adjusts its position, it can return to step S101 and start reading the return environment image at the next moment until the result that the translation amount is equal to or less than the first translation threshold is obtained.

[0084] If the rotation amount is equal to or less than the first rotation threshold and the translation amount is equal to or less than the first translation threshold, the agent has reached the position marked by the last element in the reference image feature point queue. At this time, the last element in the reference image feature point queue can be discarded and the reference image feature point queue can be updated, and then return to step S101 to start reading the return environment image at the next moment. Repeat the above process until the agent reaches the position marked by the first element in the reference image feature point queue (i.e., the feature point of the first return environment image read at time t1 in step S200), thereby achieving return.

[0085] Figure 2a A schematic flow chart of an intelligent agent autonomous tracking and returning method according to another embodiment of the present invention is shown. Figure 2a Shown is a schematic flow chart of an autonomous tracking and returning method for an intelligent agent according to another embodiment of the present invention during the intelligent agent's return stage. Figure 2b FIG. 1 is a flow chart of a method for generating a reference image feature point queue in another embodiment of the present invention at the agent outbound stage. Figure 2a and Figure 2b In the embodiment of the present invention, the intelligent agent is a drone and the image acquisition device is a camera sensor that shoots downward.

[0086] like Figure 2a As shown, in the outbound stage of the UAV, the method for generating the reference image feature point queue includes the following steps:

[0087] S1: Read the aerial image.

[0088] S2: Get the i-th aerial image.

[0089] S3: Mask the aerial image i. The masking here can be masking the landing gear in the aerial image. Since the landing gear of the drone in the aerial image does not carry any position information, it is interference information. After masking, the noise can be filtered out and the accuracy can be improved.

[0090] S4: Obtain the masked aerial image i.

[0091] S5: Determine whether i is equal to 0. If so, the masked aerial image i is the first outbound environment image, and the process proceeds to steps S6-S8. Otherwise, the masked aerial image i is not the first outbound environment image, and the process proceeds to step S9.

[0092] S6: Set the masked aerial image i as a reference aerial image and save its GNSS data.

[0093] S7: The SuperPoint feature detection algorithm is used to extract feature points of the reference aerial image.

[0094] S8: storing the feature points of the reference aerial image into a reference aerial image queue.

[0095] S9: Determine whether to skip the masked aerial image i. In S9, it can be determined whether the time interval between the masked aerial image i and the masked aerial image i-1 of the previous moment meets the interval threshold. If so, enter step S10, otherwise, return to step S1 to start reading the aerial image i+1 of the next moment. Optionally, the interval threshold in this embodiment can be determined by the time interval between each element in the reference image feature point queue. For example, if the minimum value of the time interval between each element in the reference image feature point queue is 10 seconds, the interval threshold can be set to be greater than 10 seconds. Considering that the movement of the aircraft between two consecutive images is generally a straight line or a simple rotation around the center, if every frame of the image on the outbound journey is calculated, the amount of calculation is too large and unnecessary. Therefore, this embodiment proposes a selective image extraction algorithm based on inertia, which greatly reduces the complexity of the algorithm.

[0096] S10: extract feature points of the masked aerial image i using the SuperPoint feature detection algorithm.

[0097] S11: Use the SuperGlue image matching algorithm to match the feature points of the masked aerial image i with the feature points of the latest reference aerial image.

[0098] S12: Calculate the rotation amount and / or translation amount based on the matched feature point pairs.

[0099] S13: Determine whether the rotation amount exceeds the first rotation threshold or the translation amount exceeds the first translation threshold. If the rotation amount exceeds the first rotation threshold or the translation amount exceeds the first translation threshold, proceed to steps S6-S8, otherwise proceed to step S1 to start reading the aerial image i+1 at the next moment.

[0100] like Figure 2a As shown, during the return phase of the drone, the autonomous tracking and returning method of the drone includes the following steps:

[0101] S10: Reading aerial images.

[0102] S20: Obtain the i-th aerial image.

[0103] S30: Masking the aerial image i. The masking here may be masking the landing gear in the aerial image. Since the landing gear of the drone in the aerial image does not carry any position information, it is interference information. After masking, the noise can be filtered out and the accuracy can be improved.

[0104] S40: Obtain a masked aerial image i.

[0105] S50: extracting feature points of the masked aerial image i using the SuperPoint feature detection algorithm.

[0106] S60: Using the SuperGlue image matching algorithm, the feature points of the masked aerial image i are matched with the feature points of the latest reference aerial image.

[0107] S70: Calculate the rotation amount and / or translation amount according to the matched feature point pairs.

[0108] S80: Determine whether the rotation amount exceeds the second rotation threshold or the translation amount exceeds the second translation threshold. If the rotation amount exceeds the second rotation threshold, the rotation amount is returned to form an aircraft rotation instruction. If the translation amount exceeds the second translation threshold, the translation amount is returned to form an aircraft translation instruction. If the rotation amount does not exceed the second rotation threshold and the translation amount does not exceed the second translation threshold, the last element in the reference image feature point queue is discarded and the reference image feature point queue is updated, and then return to step S10 to start reading the aerial image i+1 at the next moment.

[0109] In some embodiments, in the outbound stage or the return stage of the intelligent agent, the above-mentioned extraction of feature points of the outbound environment image or the return environment image includes: extracting feature points of the central rectangular area of ​​the outbound environment image or the return environment image. The main features of the outbound environment image or the return environment image are concentrated in the central area, so by selecting the central rectangular area of ​​the outbound environment image or the return environment image for feature extraction, the computing power requirement can be further reduced.

[0110] Compared with the prior art, the embodiments of the present invention can achieve at least one of the following beneficial effects:

[0111] 1. The solution of the present invention only requires an image acquisition device for taking images, and does not use the visual inertial odometry (VIO) algorithm and the back-end optimization steps in the simultaneous localization and mapping (SLAM) algorithm. Therefore, the algorithm complexity is lower than that of the visual inertial odometry calculation method, and the computing power requirement is relatively small. It has great application value in small intelligent entities (such as small drones) with limited computing power resources.

[0112] 2. The solution of the present invention does not use inertial gyroscope sensor data, so it is less affected by the cumulative drift, thereby improving the success rate of the intelligent body's autonomous return.

[0113] Figure 3 FIG. 2 shows an exemplary block diagram of an intelligent agent autonomous tracking and returning device according to an embodiment of the present invention. Figure 3 As shown, the intelligent agent autonomous tracking and returning device 300 includes: a first reading module 301, configured to read the return environment image collected by the image acquisition device; a first extraction and matching module 302, configured to extract the feature points of the return environment image, and match the feature points of the return environment image with the last element read from the reference image feature point queue, wherein the reference image feature point queue sequentially stores the reference image feature points of the intelligent agent at multiple moments in the outbound stage; a first calculation module 303, configured to calculate the rotation amount and / or translation amount of the intelligent agent according to the matching result; a judgment and processing module 304, configured to judge whether the rotation amount of the intelligent agent exceeds the first rotation threshold and whether the translation amount of the intelligent agent exceeds the first translation threshold, if the rotation amount of the intelligent agent exceeds the first rotation threshold or the translation amount of the intelligent agent exceeds the first translation threshold, then generate an intelligent agent rotation instruction or translation instruction according to the rotation amount or the translation amount, remove the last element from the reference image feature point queue and call the first reading module; otherwise, directly remove the last element from the reference image feature point queue and call the first reading module.

[0114] It should be understood that Figure 3 The various modules of the apparatus 300 shown in FIG. Figure 1a and Figure 1b The steps in the described method correspond to each other. Therefore, the operations, features and advantages described above for the method are also applicable to the device 300 and the modules included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0115] Figure 4 4 is a schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure. Figure 4, a structural block diagram of an electronic device 400 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein. Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404. Multiple components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the agent autonomous tracking and return method. For example, in some embodiments, the agent autonomous tracking and return method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM402 and / or the communication unit 409. When the computer program is loaded into RAM403 and executed by the computing unit 401, one or more steps of the gift package recommendation method described above may be executed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the agent autonomous tracking and returning method in any other appropriate manner (eg, by means of firmware).

[0117] Compared with the prior art, the embodiments of the present invention can achieve at least one of the following beneficial effects:

[0118] 1. The solution of the present invention only requires an image acquisition device for taking images, and does not use the visual inertial odometry (VIO) algorithm and the back-end optimization steps in the simultaneous localization and mapping (SLAM) algorithm. Therefore, the algorithm complexity is lower than that of the visual inertial odometry calculation method, and the computing power requirement is relatively small. It has great application value in small intelligent entities (such as small drones) with limited computing power resources.

[0119] 2. The solution of the present invention does not use inertial gyroscope sensor data, so it is less affected by the cumulative drift, thereby improving the success rate of the intelligent body's autonomous return.

[0120] An embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the intelligent body autonomous tracking and returning method described in any embodiment of the present invention.

[0121] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for autonomous tracking and returning of an intelligent agent, characterized in that: The following steps are performed during the agent's return phase: Step S101: reading the return environment image acquired by the image acquisition device; Step S102: extracting feature points of the return environment image, and matching the feature points of the return environment image with the last element read from the reference image feature point queue, wherein the reference image feature point queue sequentially stores reference image feature points of the agent at multiple moments in the outbound phase; Step S103: Calculate the rotation and translation of the intelligent body according to the matching results; Step S104: Determine whether the rotation amount of the intelligent body exceeds the first rotation threshold and whether the translation amount of the intelligent body exceeds the first translation threshold. If the rotation amount of the intelligent body exceeds the first rotation threshold or the translation amount of the intelligent body exceeds the first translation threshold, generate an intelligent body rotation instruction or a translation instruction according to the rotation amount or the translation amount, remove the last element from the reference image feature point queue and return to step S101; otherwise, directly remove the last element from the reference image feature point queue and return to step S101.

2. The method according to claim 1, characterized in that In the agent outbound phase, the reference image feature point queue is obtained by the following method: Step S100: reading the outbound environment image collected by the image acquisition device; Step S200: extracting feature points of the outbound environment image, and matching the feature points of the outbound environment image with the last element read from the reference image feature point queue; wherein the reference image feature point queue stores feature points of one or more reference images, and the first element of the reference image feature point queue is the feature point of the first outbound environment image; Step S300: Calculate the rotation and translation of the intelligent body according to the matching results; Step S400: Determine whether the rotation amount of the intelligent body exceeds the second rotation threshold or whether the translation amount of the intelligent body exceeds the second translation threshold. If the rotation amount of the intelligent body exceeds the second rotation threshold or the translation amount of the intelligent body exceeds the second translation threshold, set the outbound environment image as a reference image and store the feature points of the outbound environment image into the reference image feature point queue, wherein the reference image includes the satellite navigation position information of the outbound environment image; otherwise, enter step S100.

3. The method according to claim 2, characterized in that In the outbound stage or the return stage of the intelligent agent, the extracting feature points of the outbound environment image or the return environment image includes: The SuperPoint feature detection algorithm is used to extract feature points of the outbound environment image or the return environment image.

4. The method according to claim 2, characterized in that: In the outbound phase or the return phase of the agent, matching the feature points of the outbound environment image or the return environment image with the last element read from the reference image feature point queue includes: The SuperGlue image matching algorithm is used to match the feature points of the outbound environment image or the return environment image with the last element read from the reference image feature point queue.

5. The method according to claim 2, characterized in that: In the outbound stage or the return stage of the agent, before extracting the feature points of the outbound environment image or the return environment image, the following steps are also included: Determine whether the time interval between the outbound environmental image or the return environmental image and the outbound environmental image or the return environmental image at the previous moment meets the interval threshold; if so, enter the step of extracting feature points of the outbound environmental image or the return environmental image; otherwise, read the outbound environmental image or the return environmental image captured by the image acquisition device at the next moment.

6. The method according to claim 5, characterized in that The interval threshold is determined by the time interval between each element in the reference image feature point queue.

7. The method according to claim 2, characterized in that In the outbound stage or the return stage of the intelligent agent, the extracting feature points of the outbound environment image or the return environment image includes: Extract feature points of a central rectangular area of ​​the outbound environmental image or the return environmental image.

8. The method according to claim 2, characterized in that: The intelligent agent is a drone, and in the outbound stage or the return stage of the drone, before extracting the feature points of the outbound environment image or the return environment image, the following steps are also included: Mask processing is performed on the landing gear of the UAV in the outbound environmental image or the return environmental image.

9. An intelligent autonomous tracking and returning device, characterized in that: The device comprises: A first reading module is configured to read the return environment image acquired by the image acquisition device; A first extraction and matching module is configured to extract feature points of the return environment image and match the feature points of the return environment image with the last element read from a reference image feature point queue, wherein the reference image feature point queue sequentially stores reference image feature points of the agent at multiple moments in the outbound phase; A first calculation module is configured to calculate the rotation amount and / or translation amount of the intelligent body according to the matching result; The judgment and processing module is configured to judge whether the rotation amount of the intelligent body exceeds the first rotation threshold and whether the translation amount of the intelligent body exceeds the first translation threshold. If the rotation amount of the intelligent body exceeds the first rotation threshold or the translation amount of the intelligent body exceeds the first translation threshold, an intelligent body rotation instruction or a translation instruction is generated according to the rotation amount or the translation amount, and the last element is removed from the reference image feature point queue and the first reading module is called; otherwise, the last element is directly removed from the reference image feature point queue and the first reading module is called.

10. An electronic device comprising: at least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent body autonomous tracking and return method described in any one of claims 1-8.

Images