UAV target tracking method and device based on visual sensor

Abstract

This application relates to the field of image recognition technology and provides a method and device for drone target tracking based on a visual sensor. The method includes obtaining the corresponding position of a target object in each frame image preceding the nth frame image and corresponding feature information of the target object in each frame image preceding the nth frame image; the feature information includes the target object's first bounding rectangle, first RGB image, and first infrared image; determining a tracking area in the nth frame image based on each position, and determining multiple initial candidate objects in the tracking area based on each first bounding rectangle; determining a first intermediate candidate object among the multiple initial candidate objects based on each first infrared image; determining a second intermediate candidate object among the multiple initial candidate objects based on each first RGB image; and determining the target object in the nth frame image based on the first and second intermediate candidate objects. This method helps improve tracking accuracy under complex lighting conditions.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method and device for tracking a target of an unmanned aerial vehicle (UAV) based on a visual sensor. Background Art

[0002] With the rapid development of drone technology, target tracking based on visual sensors has been widely used in reconnaissance, security monitoring, agricultural inspections, and other fields. Traditional drone target tracking relies primarily on visible light cameras to obtain target information. Visible light cameras can provide rich texture and color information under good lighting conditions, but their performance degrades significantly at night, in fog, or in strong light. The limitations of visible light cameras lead to insufficient tracking accuracy, making it easy to lose or mistrack targets. Summary of the Invention

[0003] This application provides a method and device for tracking a target of an unmanned aerial vehicle (UAV) based on a visual sensor, in order to solve the problems raised by the above-mentioned background technology.

[0004] In a first aspect, the present application provides a method for tracking a target of an unmanned aerial vehicle (UAV) based on a visual sensor, comprising:

[0005] Acquire a position corresponding to the target object in each frame image before the nth frame image and feature information corresponding to the target object in each frame image before the nth frame image; the feature information includes a first circumscribed rectangle, a first RGB image, and a first infrared image of the target object;

[0006] determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles;

[0007] determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images;

[0008] determining a second intermediate candidate object from among the plurality of initial candidate objects based on each of the first RGB images;

[0009] The target object is determined in the nth frame image based on the first intermediate candidate objects and the second intermediate candidate objects.

[0010] In a possible implementation, determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles, includes:

[0011] For every two adjacent frame images before the nth frame image, determining a distance value between the target object and the two adjacent frame images based on the positions of the target object in the two adjacent frame images;

[0012] A circular area formed by taking the position of the target object in the (n-1)th frame image as the center and twice the maximum distance value among the distance values as the radius as the tracking area;

[0013] constructing a second circumscribed rectangle of each object in the tracking area, and generating a length and a width of each second circumscribed rectangle;

[0014] determining a length threshold interval and a width threshold interval based on each of the first circumscribed rectangles;

[0015] For each second circumscribed rectangle, determine whether the length of the second circumscribed rectangle is within the length threshold interval, and determine whether the width of the second circumscribed rectangle is within the width threshold interval. When the length of the second circumscribed rectangle is within the length threshold interval and the width of the second circumscribed rectangle is within the width threshold interval, determine that the object corresponding to the second circumscribed rectangle is the initial candidate object.

[0016] In a possible implementation, determining a length threshold interval and a width threshold interval based on each of the first circumscribed rectangles includes:

[0017] Determining a minimum length and a maximum length based on the lengths of the first circumscribed rectangles, and constructing the length threshold interval with three-quarters of the minimum length as the left endpoint and 1.25 times the maximum length as the right endpoint;

[0018] The minimum width and the maximum width are determined based on the widths of the first circumscribed rectangles, and the width threshold interval is constructed with three quarters of the minimum width as the left endpoint and 1.25 times the maximum width as the right endpoint.

[0019] In a possible implementation, determining a first intermediate candidate object from the plurality of initial candidate objects based on each of the first infrared images includes:

[0020] respectively acquiring a second infrared image corresponding to each of the initial objects to be selected;

[0021] Arranging the first infrared images in sequence based on a shooting time sequence of the images corresponding to the first infrared images to obtain an infrared image sequence;

[0022] For each of the initial candidate objects, generating an infrared predicted image corresponding to the initial candidate object based on the position of the target object in the (n-1)th frame image, the position of the initial candidate object in the (n)th frame image, and the infrared image sequence, and obtaining a first similarity between a second infrared image corresponding to the initial candidate object and the predicted infrared image;

[0023] An initial candidate object corresponding to the maximum first similarity is determined as the first intermediate candidate object.

[0024] In a possible implementation, generating an infrared predicted image corresponding to the initial candidate object based on the position of the target object in the (n-1)th frame image, the position of the initial candidate object in the (n)th frame image, and the infrared image sequence includes:

[0025] Calculating a target distance between a position of the target object in the (n-1)th frame image and a position of the initial candidate object in the (n)th frame image;

[0026] sequentially determining distance values between target objects corresponding to adjacent infrared images in the infrared image sequence to obtain a distance value sequence;

[0027] The infrared image sequence, the distance value sequence, the target distance and the initial candidate object are input into a preset adaptive infrared image prediction model to obtain the infrared prediction image.

[0028] In a possible implementation, determining a second intermediate candidate object from the plurality of initial candidate objects based on each of the first RGB images includes:

[0029] Obtaining a second RGB image corresponding to each of the initial objects to be selected;

[0030] For each of the initial objects to be selected, respectively obtain a second similarity between the second RGB image corresponding to the initial object to be selected and each of the first RGB images, and determine the maximum second similarity as the target second similarity corresponding to the initial object to be selected;

[0031] The initial candidate object corresponding to the maximum target second similarity is determined as the second intermediate candidate object.

[0032] In a possible implementation, determining the target object in the nth image frame based on the first intermediate candidate object and the second intermediate candidate object includes:

[0033] Determining whether the first intermediate candidate object and the second intermediate candidate object are the same object;

[0034] If they are the same object, determining the same object corresponding to the first intermediate candidate object and the second intermediate candidate object as the target object;

[0035] If they are not the same object, determining whether the second target similarity corresponding to the second intermediate candidate object is greater than a preset similarity;

[0036] If it is greater than, determining the second intermediate candidate object as the target object;

[0037] If not, the first intermediate candidate object is determined to be the target object.

[0038] In a second aspect, the present application provides a UAV target tracking device based on a visual sensor, comprising:

[0039] an acquisition module, configured to acquire a position corresponding to a target object in each frame image preceding the nth frame image and feature information corresponding to the target object in each frame image preceding the nth frame image; the feature information includes a first circumscribed rectangle, a first RGB image, and a first infrared image of the target object;

[0040] a first determining module, configured to determine a tracking area in the nth frame image based on each of the positions, and to determine a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles;

[0041] a second determining module, configured to determine a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images;

[0042] a third determining module, configured to determine a second intermediate candidate object from among the plurality of initial candidate objects based on each of the first RGB images;

[0043] The fourth determining module is configured to determine the target object in the nth frame image based on the first intermediate candidate objects and the second intermediate candidate objects.

[0044] The present application provides a method and device for tracking a target in an unmanned aerial vehicle (UAV) based on a visual sensor. The method includes: obtaining a position corresponding to a target object in each frame image preceding an nth frame image and corresponding feature information of the target object in each frame image preceding the nth frame image; the feature information includes a first bounding rectangle, a first RGB image, and a first infrared image of the target object; determining a tracking area in the nth frame image based on each of the positions, and determining multiple initial candidate objects in the tracking area based on each of the first bounding rectangles; determining a first intermediate candidate object among the multiple initial candidate objects based on each of the first infrared images; determining a second intermediate candidate object among the multiple initial candidate objects based on each of the first RGB images; and determining the target object in the nth frame image based on the first intermediate candidate object and the second intermediate candidate object. This method, on the one hand, effectively overcomes the performance limitations of traditional single visible light camera devices at night, in foggy or strong light environments by fusing the dual-modal data of RGB images and infrared images, which helps to improve the tracking accuracy under complex lighting conditions and prevent target loss or mistracking. On the other hand, it determines the tracking area and extracts the initial candidate objects based on the historical frame position information, which helps to reduce the amount of calculation. On the other hand, it independently screens the intermediate candidate objects by using the thermal radiation characteristics of the infrared image and the texture characteristics of the RGB image respectively, and determines the final target object by comprehensively comparing the intermediate screening results of the infrared and RGB dual modalities, which not only ensures the tracking accuracy but also improves the anti-interference ability in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A flowchart of a method for tracking a target using a UAV based on a visual sensor is provided in an embodiment of the present application;

[0047] Figure 2 A schematic block diagram of the structure of a UAV target tracking device based on a visual sensor provided in an embodiment of the present application;

[0048] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0051] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should be further understood that the term "and / or" used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0054] See also Figure 1 , Figure 1 The flowchart of the method for tracking a target by a UAV based on a visual sensor provided in the embodiment of the present application is as follows: Figure 1 As shown, the UAV target tracking method based on visual sensors provided in the embodiment of the present application includes steps S1 to S5.

[0055] Step S1, obtaining the corresponding position of the target object in each frame image before the nth frame image and the corresponding feature information of the target object in each frame image before the nth frame image; the feature information includes the first circumscribed rectangle, the first RGB image and the first infrared image of the target object.

[0056] Step S2: determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles.

[0057] Step S3: determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images.

[0058] Step S4: determining a second intermediate candidate object from the plurality of initial candidate objects based on each of the first RGB images.

[0059] Step S5: Determine the target object in the nth frame image based on the first intermediate candidate objects and the second intermediate candidate objects.

[0060] It should be noted that the execution entity of this embodiment can be a server or a drone target tracking device based on a visual sensor.

[0061] This embodiment specifically includes:

[0062] As described in step S1 above, the position of the target object in each frame image before the nth frame image and the corresponding feature information of the target object in each frame image before the nth frame image are obtained; specifically, the position of the target object in each frame image before the nth frame image and the corresponding feature information of the target object in each frame image before the nth frame image are obtained from a preset database;

[0063] As described in step S2 above, a tracking area is determined in the nth frame image based on each of the positions, and multiple initial candidate objects are determined in the tracking area based on each of the first bounding rectangles. Specifically, step S2 includes: for each of two adjacent frame images before the nth frame image, determining a distance value between the target object and the target object based on the position of the target object in the two adjacent frame images; defining a circular area with the position of the target object in the (n-1)th frame image as the center and a radius twice the maximum distance value among the distance values as the tracking area; constructing a second bounding rectangle for each object in the tracking area, and generating a length and width of each second bounding rectangle; determining a length threshold interval and a width threshold interval based on each of the first bounding rectangles; for each of the second bounding rectangles, determining whether the length of the second bounding rectangle is within the length threshold interval and whether the width of the second bounding rectangle is within the width threshold interval; and if the length of the second bounding rectangle is within the length threshold interval and the width of the second bounding rectangle is within the width threshold interval, determining the object corresponding to the second bounding rectangle as the initial candidate object.

[0064] As described in step S3 above, a first intermediate candidate object is determined from the plurality of initial candidate objects based on each of the first infrared images. Specifically, step S3 includes: obtaining a second infrared image corresponding to each of the initial candidate objects; arranging the first infrared images in sequence based on the shooting time sequence of the images corresponding to the first infrared images to obtain an infrared image sequence; for each initial candidate object, generating an infrared predicted image corresponding to the initial candidate object based on the position of the target object in the n-1th frame image, the position of the initial candidate object in the nth frame image, and the infrared image sequence; obtaining a first similarity between the second infrared image corresponding to the initial candidate object and the predicted infrared image; and determining the initial candidate object corresponding to the maximum first similarity as the first intermediate candidate object.

[0065] As described in step S4 above, a second intermediate candidate object is determined from the plurality of initial candidate objects based on each of the first RGB images. Specifically, step S4 includes: obtaining a second RGB image corresponding to each of the initial candidate objects; obtaining, for each of the initial candidate objects, a second similarity between the second RGB image corresponding to the initial candidate object and each of the first RGB images, and determining the largest second similarity as the target second similarity corresponding to the initial candidate object; and determining the initial candidate object corresponding to the largest target second similarity as the second intermediate candidate object.

[0066] As described in step S5 above, the target object is determined in the nth frame of image based on the first intermediate candidate object and the second intermediate candidate object. Specifically, step S5 includes: determining whether the first intermediate candidate object and the second intermediate candidate object are the same object; if so, determining the same object corresponding to the first intermediate candidate object and the second intermediate candidate object as the target object; if not, determining whether a second target similarity corresponding to the second intermediate candidate object is greater than a preset similarity; if so, determining the second intermediate candidate object as the target object; if not, determining the first intermediate candidate object as the target object.

[0067] The method provided in this embodiment, on the one hand, effectively overcomes the performance limitations of traditional single visible light camera devices at night, in foggy conditions, or in strong light environments by fusing the dual-modal data of RGB images and infrared images, which helps to improve tracking accuracy under complex lighting conditions and prevent target loss or mistracking. On the other hand, it determines the tracking area and extracts the initial candidate objects based on historical frame position information, which helps to reduce the amount of calculation. On the other hand, it independently screens intermediate candidate objects by using the thermal radiation characteristics of the infrared image and the texture characteristics of the RGB image respectively, and determines the final target object by comprehensively comparing the intermediate screening results of the infrared and RGB dual modalities, which not only ensures tracking accuracy but also improves the anti-interference ability in complex scenes.

[0068] In some embodiments, determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles, comprises the following steps:

[0069] For every two adjacent frame images before the nth frame image, determining a distance value between the target object and the two adjacent frame images based on the positions of the target object in the two adjacent frame images;

[0070] A circular area formed by taking the position of the target object in the (n-1)th frame image as the center and twice the maximum distance value among the distance values as the radius as the tracking area;

[0071] constructing a second circumscribed rectangle of each object in the tracking area, and generating a length and a width of each second circumscribed rectangle;

[0072] determining a length threshold interval and a width threshold interval based on each of the first circumscribed rectangles;

[0073] For each second circumscribed rectangle, determine whether the length of the second circumscribed rectangle is within the length threshold interval, and determine whether the width of the second circumscribed rectangle is within the width threshold interval. When the length of the second circumscribed rectangle is within the length threshold interval and the width of the second circumscribed rectangle is within the width threshold interval, determine that the object corresponding to the second circumscribed rectangle is the initial candidate object.

[0074] The method provided in this embodiment, on the one hand, calculates the tracking area with the maximum displacement range by analyzing the movement distance of the target object between adjacent frames. Compared with the method of fixing the search area, it can effectively cover the range where the target may appear and avoid the waste of computing resources caused by global search. On the other hand, a circular tracking area is set up, radiating outward from the position of the target in the previous frame as the center. This design not only conforms to the uncertainty characteristics of target motion, but also ensures the integrity of the search area. On the other hand, by constructing the length and width threshold range of the target's historical circumscribed rectangle, precise size constraints are provided for candidate object screening, which can eliminate interference objects with excessive size differences.

[0075] In some embodiments, determining a length threshold interval and a width threshold interval based on each of the first circumscribed rectangles includes the following steps:

[0076] Determining a minimum length and a maximum length based on the lengths of the first circumscribed rectangles, and constructing the length threshold interval with three-quarters of the minimum length as the left endpoint and 1.25 times the maximum length as the right endpoint;

[0077] The minimum width and the maximum width are determined based on the widths of the first circumscribed rectangles, and the width threshold interval is constructed with three quarters of the minimum width as the left endpoint and 1.25 times the maximum width as the right endpoint.

[0078] The method provided in this embodiment, on the one hand, establishes a dynamic threshold range based on the actual size changes of the target by counting the extreme length and width of the target's circumscribed rectangle in historical frames. This is more adaptable to changes in target scale than a fixed threshold. On the other hand, three-quarters of the minimum value is used as the lower limit of the interval, which not only takes into account the reasonable range within which the target may shrink, but also avoids false exclusion due to measurement errors. On the other hand, 1.25 times the maximum value is used as the upper limit of the interval, leaving sufficient space for possible target enlargement, ensuring that tracking will not be lost due to temporary enlargement of the target. This interval design fully considers the deformation characteristics that may occur in the target object during movement. While ensuring screening accuracy, it greatly improves the system's adaptability to target scale changes and provides a reliable size screening benchmark for subsequent precise tracking.

[0079] In some embodiments, determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images comprises the following steps:

[0080] respectively acquiring a second infrared image corresponding to each of the initial objects to be selected;

[0081] Arranging the first infrared images in sequence based on a shooting time sequence of the images corresponding to the first infrared images to obtain an infrared image sequence;

[0082] For each of the initial candidate objects, an infrared predicted image corresponding to the initial candidate object is generated based on the position of the target object in the (n-1)th frame image, the position of the initial candidate object in the (n)th frame image, and the infrared image sequence, and a first similarity between a second infrared image corresponding to the initial candidate object and the predicted infrared image is obtained; specifically, the second infrared image and the predicted infrared image are respectively input into a preset infrared image feature extraction model to obtain a first feature vector and a second feature vector, and a cosine value between the first feature vector and the second feature vector is determined as the first similarity;

[0083] An initial candidate object corresponding to the maximum first similarity is determined as the first intermediate candidate object.

[0084] The step of generating an infrared predicted image corresponding to the initial candidate object based on the position of the target object in the n-1th frame image, the position of the initial candidate object in the nth frame image, and the infrared image sequence includes the following steps:

[0085] Calculating a target distance between a position of the target object in the (n-1)th frame image and a position of the initial candidate object in the (n)th frame image;

[0086] Sequentially determining distance values between target objects corresponding to adjacent infrared images in the infrared image sequence to obtain a distance value sequence; specifically, for two adjacent infrared images, determining the distance value between target objects corresponding to the two adjacent infrared images based on a position of the target object corresponding to a preceding infrared image and a position of the target object corresponding to a succeeding infrared image;

[0087] The infrared image sequence, the distance value sequence, the target distance and the initial candidate object are input into a preset adaptive infrared image prediction model to obtain the infrared prediction image; wherein, after receiving the infrared image sequence, the distance value sequence, the target distance and the initial candidate object, the adaptive infrared image prediction model first learns the relationship between the surface temperature change and the distance value of the target object based on the infrared image sequence and the distance value sequence to obtain the correlation between the surface temperature change and the distance value of the target object, and then renders the initial candidate object based on the learned correlation and the target distance to obtain the infrared prediction image.

[0088] The method provided in this embodiment, on the one hand, establishes a correlation model between target temperature change and movement distance by constructing a time-series infrared image sequence and analyzing the historical movement characteristics of the target, so that the system can accurately predict the infrared characteristics that the target may present in the current frame. On the other hand, an adaptive infrared image prediction model is adopted to realize intelligent rendering of the infrared characteristics of the initial candidate object by learning the correlation between the target surface temperature change and the movement distance in historical data. On the other hand, by calculating the similarity between the actual infrared image of the initial candidate object and the predicted infrared image, heat source interference objects can be distinguished.

[0089] In some embodiments, determining a second intermediate candidate object from the plurality of initial candidate objects based on each of the first RGB images comprises the following steps:

[0090] Obtaining the second RGB image corresponding to each of the initial objects to be selected respectively;

[0091] For each of the initial objects to be selected, respectively obtain a second similarity between a second RGB image corresponding to the initial object to be selected and each of the first RGB images, and determine the maximum second similarity as a target second similarity corresponding to the initial object to be selected; specifically, input the second RGB image and the first RGB image into a preset RGB image feature extraction model, respectively, to obtain a third eigenvector and a fourth eigenvector, and determine the cosine value between the third eigenvector and the fourth eigenvector as the second similarity;

[0092] The initial candidate object corresponding to the maximum target second similarity is determined as the second intermediate candidate object.

[0093] The method provided in this embodiment, on the one hand, for each of the initial candidate objects, by determining the maximum second similarity among the second similarities corresponding to the initial candidate objects as the target second similarity corresponding to the initial candidate object, takes into account possible multi-angle feature changes of the target object and ensures the acquisition of the best matching result. On the other hand, by determining the initial candidate object corresponding to the maximum target second similarity as the second intermediate candidate object, an accurate RGB feature basis is provided for subsequent multimodal decision-making.

[0094] In some embodiments, determining the target object in the nth frame image based on the first intermediate candidate object and the second intermediate candidate object includes the following steps:

[0095] Determining whether the first intermediate candidate object and the second intermediate candidate object are the same object; specifically, if the positions of the first intermediate candidate object and the second intermediate candidate object are consistent, and the contours of the first intermediate candidate object and the second intermediate candidate object are consistent, determining that the first intermediate candidate object and the second intermediate candidate object are the same object;

[0096] If they are the same object, determining the same object corresponding to the first intermediate candidate object and the second intermediate candidate object as the target object;

[0097] If they are not the same object, determining whether the second target similarity corresponding to the second intermediate candidate object is greater than a preset similarity;

[0098] If it is greater than, determining the second intermediate candidate object as the target object;

[0099] If not, the first intermediate candidate object is determined to be the target object.

[0100] The method provided in this embodiment, on the one hand, determines the consistency of the infrared and RGB modal screening results through a dual verification mechanism of position and contour, thereby ensuring the rigor of multimodal data fusion. On the other hand, when the dual-modal results point to the same object, the target is directly confirmed. This collaborative verification mechanism significantly improves the accuracy of target recognition in complex environments. On the other hand, by constructing a progressive decision-making process of "collaborative verification first, then confidence selection", it fully utilizes the advantages of multimodal fusion and ensures the system's fault tolerance when the RGB modality fails, so that the UAV target tracking system can maintain stable and reliable tracking performance in various complex environments.

[0101] See also Figure 2 , Figure 2 The schematic block diagram of the structure of the UAV target tracking device 100 based on the visual sensor provided in the embodiment of the present application is as follows: Figure 2 As shown, the UAV target tracking device 100 based on a visual sensor provided in an embodiment of the present application includes:

[0102] The acquisition module 110 is used to obtain the corresponding position of the target object in each frame image before the nth frame image and the corresponding feature information of the target object in each frame image before the nth frame image; the feature information includes the first circumscribed rectangle, the first RGB image and the first infrared image of the target object.

[0103] The first determining module 120 is configured to determine a tracking area in the nth frame image based on the positions, and determine a plurality of initial objects to be selected in the tracking area based on the first circumscribed rectangles.

[0104] The second determining module 130 is configured to determine a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images.

[0105] The third determining module 140 is configured to determine a second intermediate candidate object from among the plurality of initial candidate objects based on each of the first RGB images.

[0106] The fourth determining module 150 is configured to determine the target object in the nth frame image based on the first intermediate candidate objects and the second intermediate candidate objects.

[0107] It should be noted that, those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the above-described system and each module can refer to the processes in the aforementioned embodiment of the UAV target tracking method based on visual sensors, and will not be repeated here.

[0108] The UAV target tracking device 100 based on the visual sensor provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 3 The system runs on the terminal device 200 shown.

[0109] See also Figure 3 , Figure 3 This is a schematic block diagram of the structure of a terminal device 200 provided in an embodiment of the present application. The terminal device 200 includes a processor 201 and a memory 202. The processor 201 and the memory 202 are connected via a system bus 203, wherein the memory 202 may include a non-volatile storage medium and an internal memory.

[0110] The non-volatile storage medium may store a computer program. The computer program includes program instructions, which, when executed by the processor 201, may cause the processor 201 to execute any of the above-mentioned UAV target tracking methods based on a visual sensor.

[0111] The processor 201 is used to provide computing and control capabilities to support the operation of the entire terminal device 200.

[0112] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 201, the processor 201 can execute any of the above-mentioned drone target tracking methods based on visual sensors.

[0113] Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the terminal device 200 involved in the solution of the present application. The specific terminal device 200 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0114] It should be understood that the processor 201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0115] In some embodiments, the processor 201 is configured to execute a computer program stored in the memory to implement the following steps:

[0116] Acquire a position corresponding to the target object in each frame image before the nth frame image and feature information corresponding to the target object in each frame image before the nth frame image; the feature information includes a first circumscribed rectangle, a first RGB image, and a first infrared image of the target object;

[0117] determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles;

[0118] determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images;

[0119] determining a second intermediate candidate object from among the plurality of initial candidate objects based on each of the first RGB images;

[0120] The target object is determined in the nth frame image based on the first intermediate candidate objects and the second intermediate candidate objects.

[0121] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the terminal device 200 described above can refer to the corresponding process of the aforementioned drone target tracking method based on visual sensors, and will not be repeated here.

[0122] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the drone target tracking method based on a visual sensor as provided in an embodiment of the present application.

[0123] The computer-readable storage medium may be an internal storage unit of the terminal device 200 in the aforementioned embodiment, such as a hard disk or memory of the terminal device 200. The computer-readable storage medium may also be an external storage device of the terminal device 200, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped with the terminal device 200.

[0124] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A UAV target tracking method based on a visual sensor, characterized in that: include: Obtaining the corresponding position of the target object in each frame image before the nth frame image and the corresponding feature information of the target object in each frame image before the nth frame image; The feature information includes a first circumscribed rectangle, a first RGB image, and a first infrared image of the target object; determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles; determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images; determining a second intermediate candidate object from among the plurality of initial candidate objects based on each of the first RGB images; determining the target object in the nth frame of image based on the first intermediate candidate objects and the second intermediate candidate objects; The step of determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images includes: respectively acquiring a second infrared image corresponding to each of the initial objects to be selected; Arranging the first infrared images in sequence based on a shooting time sequence of the images corresponding to the first infrared images to obtain an infrared image sequence; For each of the initial candidate objects, generating an infrared predicted image corresponding to the initial candidate object based on the position of the target object in the (n-1)th frame image, the position of the initial candidate object in the (n)th frame image, and the infrared image sequence, and obtaining a first similarity between a second infrared image corresponding to the initial candidate object and the predicted infrared image; Determine the initial candidate object corresponding to the maximum first similarity as the first intermediate candidate object; The determining of a second intermediate candidate object from the plurality of initial candidate objects based on each of the first RGB images includes: Obtaining a second RGB image corresponding to each of the initial objects to be selected; For each of the initial objects to be selected, respectively obtain a second similarity between the second RGB image corresponding to the initial object to be selected and each of the first RGB images, and determine the maximum second similarity as the target second similarity corresponding to the initial object to be selected; The initial candidate object corresponding to the maximum target second similarity is determined as the second intermediate candidate object.

2. The method for tracking a target by a UAV based on a visual sensor according to claim 1, wherein: The step of determining a tracking area in the nth frame image based on each of the positions, and determining a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles, includes: For every two adjacent frame images before the nth frame image, determining a distance value between the target object and the two adjacent frame images based on the positions of the target object in the two adjacent frame images; A circular area formed by taking the position of the target object in the (n-1)th frame image as the center and twice the maximum distance value among the distance values as the radius as the tracking area; constructing a second circumscribed rectangle of each object in the tracking area, and generating a length and a width of each second circumscribed rectangle; determining a length threshold interval and a width threshold interval based on each of the first circumscribed rectangles; For each second circumscribed rectangle, determine whether the length of the second circumscribed rectangle is within the length threshold interval, and determine whether the width of the second circumscribed rectangle is within the width threshold interval. When the length of the second circumscribed rectangle is within the length threshold interval and the width of the second circumscribed rectangle is within the width threshold interval, determine that the object corresponding to the second circumscribed rectangle is the initial candidate object.

3. The method for tracking a target by a UAV based on a visual sensor according to claim 2, wherein: The determining of a length threshold interval and a width threshold interval based on each of the first circumscribed rectangles includes: Determining a minimum length and a maximum length based on the lengths of the first circumscribed rectangles, and constructing the length threshold interval with three-quarters of the minimum length as the left endpoint and 1.25 times the maximum length as the right endpoint; The minimum width and the maximum width are determined based on the widths of the first circumscribed rectangles, and the width threshold interval is constructed with three quarters of the minimum width as the left endpoint and 1.25 times the maximum width as the right endpoint.

4. The method for tracking a target by a UAV based on a visual sensor according to claim 1, wherein: The generating of the infrared prediction image corresponding to the initial candidate object based on the position of the target object in the (n-1)th frame image, the position of the initial candidate object in the (n)th frame image, and the infrared image sequence includes: Calculating a target distance between a position of the target object in the (n-1)th frame image and a position of the initial candidate object in the (n)th frame image; sequentially determining distance values between target objects corresponding to adjacent infrared images in the infrared image sequence to obtain a distance value sequence; The infrared image sequence, the distance value sequence, the target distance and the initial candidate object are input into a preset adaptive infrared image prediction model to obtain the infrared prediction image.

5. The method for tracking a target by a UAV based on a visual sensor according to claim 1, wherein: The determining the target object in the nth frame image based on the first intermediate candidate object and the second intermediate candidate object includes: Determining whether the first intermediate candidate object and the second intermediate candidate object are the same object; If they are the same object, determining the same object corresponding to the first intermediate candidate object and the second intermediate candidate object as the target object; If they are not the same object, determining whether the second target similarity corresponding to the second intermediate candidate object is greater than a preset similarity; If it is greater than, determining the second intermediate candidate object as the target object; If not, the first intermediate candidate object is determined to be the target object.

6. A UAV target tracking device based on a visual sensor, characterized in that: include: An acquisition module is used to acquire the corresponding position of the target object in each frame image before the nth frame image and the corresponding feature information of the target object in each frame image before the nth frame image; The feature information includes a first circumscribed rectangle, a first RGB image, and a first infrared image of the target object; a first determining module, configured to determine a tracking area in the nth frame image based on each of the positions, and to determine a plurality of initial objects to be selected in the tracking area based on each of the first circumscribed rectangles; a second determining module, configured to determine a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images; a third determining module, configured to determine a second intermediate candidate object from among the plurality of initial candidate objects based on each of the first RGB images; a fourth determining module, configured to determine the target object in the nth frame of image based on the first intermediate candidate objects and the second intermediate candidate objects; The step of determining a first intermediate candidate object from among the plurality of initial candidate objects based on each of the first infrared images includes: respectively acquiring a second infrared image corresponding to each of the initial objects to be selected; Arranging the first infrared images in sequence based on a shooting time sequence of the images corresponding to the first infrared images to obtain an infrared image sequence; For each of the initial candidate objects, generating an infrared predicted image corresponding to the initial candidate object based on the position of the target object in the (n-1)th frame image, the position of the initial candidate object in the (n)th frame image, and the infrared image sequence, and obtaining a first similarity between a second infrared image corresponding to the initial candidate object and the predicted infrared image; Determine the initial candidate object corresponding to the maximum first similarity as the first intermediate candidate object; The determining of a second intermediate candidate object from the plurality of initial candidate objects based on each of the first RGB images includes: Obtaining the second RGB image corresponding to each of the initial objects to be selected respectively; For each of the initial objects to be selected, respectively obtain a second similarity between the second RGB image corresponding to the initial object to be selected and each of the first RGB images, and determine the maximum second similarity as the target second similarity corresponding to the initial object to be selected; The initial candidate object corresponding to the maximum target second similarity is determined as the second intermediate candidate object.

Images