Target detection tracking method, device and equipment based on aircraft and medium

Abstract

The invention relates to an aircraft-based target detection tracking method, device and equipment and a medium, and the method comprises the steps: firstly obtaining a first image of an aircraft, and then carrying out the motion compensation to obtain a second image; extracting a plurality of first pixel positions from the first image, extracting a plurality of second pixel positions from the second image, and mapping a second detection frame for extracting the plurality of second pixel positions from the second image into the first image, the first detection frame or the second detection frame with the similarity between the first detection frame and the second detection frame higher than the target threshold value is reserved, the first detection frame and the second detection frame with the similarity lower than the target threshold value are smoothed, and a third image converted from the first image is obtained, so that the accuracy of detecting the target object to be tracked from the third image is improved; and finally, training a preset tracker by using the third image so as to improve the tracking accuracy of the trained tracker on the to-be-tracked target object in the image of the aircraft.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of target detection technology, and in particular, to a target detection and tracking method, device, equipment and medium based on an aircraft. Background Art

[0002] Autonomous tracking of aircraft targets is a top priority research direction for many scientific researchers.

[0003] At present, target detection and target tracking of high-speed aircraft mostly use high-definition cameras, which transmit the camera images in real time to the ground terminal for operators to perform target identification, target confirmation, target locking, and target tracking. This tracking and detection process is cumbersome and requires many manual intervention links, which can easily lead to missed detection of targets. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a method, device, equipment and medium for target detection and tracking based on an aircraft, which can improve the accuracy of detecting a target object to be tracked from an image, thereby achieving more accurate target tracking.

[0006] A first aspect of an embodiment of the present application provides a target detection and tracking method based on an aircraft, the method comprising:

[0007] Acquire a first image; the first image is an image taken by the aircraft at a first moment by a corresponding photographing device during flight;

[0008] According to the linear variation law of pixel points in at least two consecutive frames of images captured by the shooting device corresponding to the flight trajectory of the aircraft, motion compensation is performed on the first image to obtain a second image;

[0009] Detecting a plurality of first pixel positions of a target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions;

[0010] Detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0011] According to the linear variation rule, mapping the plurality of second pixel positions and the second detection frame in the second image into the first image;

[0012] Calculating a similarity between the first detection frame and the second detection frame in the first image;

[0013] Draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame smoothed according to the first detection frame and the second detection frame when the similarity is less than or equal to the target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity is greater than the target threshold.

[0014] This embodiment has at least the following beneficial effects:

[0015] This embodiment performs motion compensation on the first image to eliminate image differences caused by the linear change law of the device moving with the aircraft, reduce interference, and improve detection accuracy; then, according to the linear change law corresponding to the pixel points in the continuous image calculated above and the flight trajectory of the aircraft, the second detection frame of the multiple second pixel positions extracted from the second image is mapped to the first image, and the similarity between the first detection frame and the second detection frame of the multiple first pixel positions extracted from the first image is calculated; then, the first detection frame or the second detection frame above the target threshold of the similarity is retained, and the first detection frame and the second detection frame below the target threshold of the similarity are smoothed to obtain a third detection frame, and then the third detection frame is mapped to the first image to obtain a third image, so as to improve the accuracy of detecting the target object to be tracked from the third image, and finally, the preset tracker is trained using the third image to improve the tracking accuracy of the trained tracker on the target object to be tracked in the image taken by the aircraft.

[0016] Detecting and tracking the target object to be tracked can improve the accuracy of detecting and tracking the target object to be tracked.

[0017] In some embodiments of the present application, calculating the similarity between the first detection frame and the second detection frame in the first image captured at the first moment includes:

[0018] Calculate a first central pixel in the first detection frame in the first image;

[0019] Calculate a second central pixel in the second detection frame in the first image;

[0020] determining a Euclidean distance between the first central pixel and the second central pixel;

[0021] The similarity between the first detection frame and the second detection frame is calculated according to the Euclidean distance.

[0022] In some embodiments of the present application, the first detection frame and the second detection frame are smoothed by the following formula to obtain a third detection frame:

[0023]

[0024] Where s is the target threshold, is the first detection frame, is the second detection frame, is the third detection frame.

[0025] In some embodiments of the present application, the step of training a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the capturing device at the second moment according to the trained tracker, includes:

[0026] Extracting pixel features of the target object to be tracked in the third image;

[0027] Training a preset tracker according to the pixel features to obtain a trained tracker; the tracker is a classifier;

[0028] Determining the position of the target object to be tracked in the image captured by the shooting device at the second moment according to the trained tracker; wherein the trained tracker determines the position of the target object to be tracked in the image captured by the shooting device at the second moment, comprising:

[0029] Performing Gaussian kernel function dot multiplication on the pixel features in the trained tracker to obtain a response map output by the trained tracker;

[0030] The position with the largest response value is found in the response map, and the maximum position is used as the detected position of the target object to be tracked in the image captured by the shooting device at the second moment.

[0031] In some embodiments of the present application, the trained tracker performs a Gaussian kernel function dot multiplication on the pixel features in the frequency domain.

[0032] In some embodiments of the present application, performing motion compensation on the first image according to the linear change law of pixel points in at least two consecutive frames of images captured by the shooting device corresponding to the flight trajectory of the aircraft includes:

[0033] Acquire at least two frames of continuous images captured by the shooting device;

[0034] Selecting a pixel point to be tracked in a first frame of the at least two consecutive frames of images;

[0035] Determine the position of the pixel point to be tracked in the images of other frames of the at least two consecutive images except the first frame;

[0036] Determining the motion trajectory of the pixel to be tracked according to the position;

[0037] According to the motion trajectory, a global motion model is constructed to characterize the linear change law of pixel points corresponding to the flight trajectory of the aircraft;

[0038] Motion compensation is performed on the first image according to the global motion model.

[0039] In some embodiments of the present application, the detecting a plurality of first pixel positions of the target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions includes:

[0040] Detecting a plurality of first pixel positions of the target object to be tracked in the first image by using a target detection model obtained based on neural network training, and detecting a first detection frame corresponding to the plurality of first pixel positions;

[0041] The detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions includes:

[0042] The target detection model is used to detect multiple second pixel positions of the target object to be tracked in the second image, and a second detection frame corresponding to the multiple second pixel positions is detected.

[0043] A second aspect of an embodiment of the present application provides a target detection and tracking device based on an aircraft, the device comprising:

[0044] An image acquisition unit, used to acquire a first image; the first image is an image taken by the aircraft at a first moment during flight by a corresponding shooting device;

[0045] a motion compensation unit, configured to perform motion compensation on the first image according to a linear change rule of pixel points in at least two consecutive frames of images captured by the capturing device corresponding to the flight trajectory of the aircraft, so as to obtain a second image;

[0046] A first detection unit, configured to detect a plurality of first pixel positions of the target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions;

[0047] A second detection unit, configured to detect a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0048] An image mapping unit, configured to map the plurality of second pixel positions and the second detection frame in the second image into the first image according to the linear variation rule;

[0049] a similarity calculation unit, configured to calculate a similarity between the first detection frame and the second detection frame in the first image;

[0050] an image generating unit, configured to draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame smoothed according to the first detection frame and the second detection frame when the similarity is less than or equal to a target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity is greater than a target threshold;

[0051] An object tracking unit is used to train a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the shooting device at a second moment according to the trained tracker, wherein the second moment is after the first moment.

[0052] The electronic device provided in the third aspect of an embodiment of the present application includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the aircraft-based target detection and tracking method.

[0053] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the aircraft-based target detection and tracking method.

[0054] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 It is a flowchart of an embodiment of an aircraft-based target detection and tracking method provided by the present application;

[0057] Figure 2is a structural schematic diagram of an embodiment of a target tracking device provided by the present application;

[0058] Figure 3 yes Figure 2 A schematic diagram of the structure of an embodiment of an aircraft-based target detection and tracking device provided by a main control module;

[0059] Figure 4 It is a schematic structural diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] like Figure 1 An embodiment of the present application provides a target detection and tracking method based on an aircraft, the method comprising the following steps:

[0064] Step S110, acquiring a first image; the first image is an image taken by the aircraft at a first moment by a corresponding shooting device during flight;

[0065] Step S120, performing motion compensation on the first image according to a linear variation rule of pixel points in at least two consecutive frames of images captured by the capturing device corresponding to the flight trajectory of the aircraft, to obtain a second image;

[0066] Step S130, detecting a plurality of first pixel positions of the target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions;

[0067] Step S140, detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0068] Step S150, mapping a plurality of second pixel positions and a second detection frame in the second image into the first image according to a linear variation rule;

[0069] Step S160, calculating the similarity between the first detection frame and the second detection frame in the first image;

[0070] Step S170, drawing a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame that is smoothed based on the first detection frame and the second detection frame when the degree of similarity is less than or equal to the target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the degree of similarity is greater than the target threshold;

[0071] Step S180, training a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the shooting device at a second moment according to the trained tracker, wherein the second moment is after the first moment.

[0072] In step S110, the aircraft may be a drone, an airplane, a rocket, an airship, etc. A shooting device is provided on the aircraft, and the shooting device may be a camera or a video camera, which is not limited here. The shooting device may shoot during the flight of the aircraft, and the image captured by the aircraft at the first moment is referred to as the first image.

[0073] It should be noted that the number of first images is not limited here, and it can be one frame of image or multiple frames of image.

[0074] In step S120, the first image captured due to the high-speed flight of the aircraft is distorted, which is not conducive to target detection and is likely to cause missed detection and false detection. Experimental analysis shows that the shooting device moves with the aircraft during the flight, so most of the pixels of the image captured by the shooting device will follow the movement pattern of the aircraft. This movement pattern of the overall movement of the scene can be described by constructing a global motion model. By estimating this global motion model, the image difference caused by the movement is predicted, and the first image is compensated to obtain the second image.

[0075] Therefore, step S120 includes the following steps:

[0076] At least two frames of continuous images captured by the camera are obtained. The continuous images here refer to continuous image frames, for example, the camera captures images at intervals to obtain continuous images, or the camera captures images and samples continuous images through a sampling algorithm.

[0077] Select the pixel point to be tracked in the first frame of the continuous image.

[0078] In some embodiments, the pixel points to be tracked are corner points and edge points. Whether a pixel point is a corner point is determined based on the grayscale change caused by moving the local window of the image in any direction. If a pixel point is in its neighborhood, no matter in which direction the window is moved, the grayscale change in the window is very large, then the point is considered to be a corner point. The calculation principle is as follows:

[0079] R=det(M)-kgtrace 2 (M) (1)

[0080] Where R is the corner response function value; det(M) is the determinant of M; trace(M) is the trace of M; k is a hyperparameter, generally ranging from 0.04 to 0.06; where the expressions of det(M) and trace(M) are as follows:

[0081]

[0082] Where I is the gradient value in the x and y directions; w(x,y) is the Gaussian weight of each pixel in the window;

[0083]

[0084] It is expressed as the sum of the elements on the diagonal of the matrix.

[0085] Determine the position of the pixel point to be tracked in other images of the continuous image except the first frame image. In this step, the corner points and edge points can be tracked to determine their motion trajectory.

[0086] According to the position, the motion trajectory of the pixel to be tracked is determined; according to the motion trajectory of the corner points and edge points, a global motion model including translation and affine change is constructed.

[0087] According to the motion trajectory, a global motion model is constructed to characterize the linear change law of pixel points corresponding to the flight trajectory of the aircraft.

[0088] Motion compensation is performed on the first image according to the global motion model.

[0089] Affine transformation is the spatial transformation of pixels in the image, that is, the linear transformation from two-dimensional coordinates to two-dimensional coordinates. The calculation process is:

[0090]

[0091] Among them, (x', y') is the coordinate after transformation; (x, y) is the original coordinate; a~f are 6 degrees of freedom, the four parameters a, b, d, and e control the linear transformation, and c and f control the translation bias.

[0092] The constructed global motion model is used to perform motion compensation on the first image, and the motion-compensated first image is encoded and transmitted to an H.265 / HEVC decoder for decoding to reconstruct the first image to obtain a second image.

[0093] In step S130, the target object to be tracked in the first image may be detected using a target detection model obtained based on neural network training, wherein the target object to be tracked is an object to be tracked. The target object to be tracked may be specified in advance, such as a drone in front of the aircraft.

[0094] For example, the shallow surface features and deep semantic features of the target object to be tracked are learned in advance based on the artificial neural network, and the model weights of the feature information of the target object to be tracked are obtained. Then, feature extraction, feature information filtering, feature attribute discrimination, non-maximum suppression and other technical means are performed on the entire first image based on the model weights to identify the target object to be tracked, and the set threshold is compared with the confidence of the identified target object to be tracked. If the confidence is higher than the set threshold, the detection is successful, and the position, label, confidence, detection box and other information of the target object to be tracked are output.

[0095] In step S140, the target detection model may also be used to detect the target object to be tracked in the second image, which will not be described in detail here.

[0096] In step S150, the relevant pixels in the second image may be mapped to the first image according to the inverse operation of the global motion model given above.

[0097] It should be noted that the first image includes a first pixel position and a corresponding first detection frame in the original first image, and a second pixel position and a corresponding second detection frame in the original second image that are mapped thereto.

[0098] In step S160, calculating the similarity between the first detection frame and the second detection frame in the first image includes the following steps:

[0099] Calculate a first central pixel in a first detection frame in the first image;

[0100] Calculate a second central pixel in a second detection frame in the first image;

[0101] determining a Euclidean distance between a first central pixel and a second central pixel;

[0102] The similarity between the first detection frame and the second detection frame is calculated according to the Euclidean distance.

[0103] In step S170, a third detection frame is drawn in the first image to obtain a third image, wherein the third detection frame is a detection frame that is smoothed based on the first detection frame and the second detection frame when the degree of similarity is less than or equal to the target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the degree of similarity is greater than the target threshold.

[0104] Specifically, the third detection frame is obtained by smoothing the first detection frame and the second detection frame using the following formula:

[0105]

[0106] Where s is the target threshold, is the first detection frame, is the second detection frame, is the third detection frame.

[0107] The present application includes a detection process and a tracking process, wherein the detection process includes the aforementioned steps, and the tracking process includes step S180;

[0108] In step S180, the steps specifically include:

[0109] Extracting pixel features in the target object to be tracked in the third image;

[0110] The preset tracker is trained according to the pixel features to obtain a trained tracker; the tracker is a classifier;

[0111] Determining the position of the target object to be tracked in the image captured by the shooting device at the second moment according to the trained tracker; wherein the trained tracker determines the position of the target object to be tracked in the image captured by the shooting device at the second moment, including:

[0112] Perform Gaussian kernel function dot multiplication on the pixel features in the trained tracker to obtain the response map output by the trained tracker;

[0113] The position with the largest response value is found in the response graph, and the largest position is used as the position of the detected target object to be tracked in the image captured by the shooting device at the second moment.

[0114] In this embodiment, the target object to be tracked is confirmed by the ground operator on the third image, for example, the third image is first transmitted to the ground, and after confirmation by the ground operator, the information of the target object to be tracked sent by the ground is received, and the pixel features of the target object to be tracked are extracted. It can also be confirmed in advance by the ground, that is, the ground operator confirms the target object to be tracked in the first image, and then receives the information of the target object to be tracked transmitted by the ground. The pixel features here include but are not limited to gradient histograms and color histograms.

[0115] This embodiment uses a discriminant classifier trained as a tracker to achieve tracking of the target object to be tracked, and uses circulant matrix and fast Fourier transform to accelerate the calculation process, thereby ensuring tracking accuracy while improving tracking speed. The specific process is as follows:

[0116] First, the training process of the tracker is introduced:

[0117] The pixel features of the sample image (i.e., the third image of multiple frames) are converted to the frequency domain using fast Fourier transform to accelerate subsequent calculations, and the response map of the classifier output is obtained by multiplying the pixel features of the target object to be tracked with the Gaussian kernel function, and then the tracker is trained using the response map to obtain the weight coefficient of the tracker;

[0118] The tracker's response is calculated as follows:

[0119]

[0120] in, is the Fourier transform of the response; is the first row of the Fourier transform of the kernel correlation matrix between x and z, are the coefficients obtained by solving the ridge regression problem, and e is the dot product;

[0121] The solution formula of ridge regression in frequency is as follows:

[0122]

[0123] in, is the Fourier transform of w, x, y; x is the training sample matrix; y is the sample label; λ is the regularization parameter;

[0124] During use, for any image captured by the aircraft, the pixel features of the target object to be tracked are also extracted from it, and the pixel features are multiplied by the Gaussian kernel function of the trained tracker to obtain a response map, and the position of the maximum response is found in the response map. This position is the position of the target object to be tracked in the image, thereby achieving tracking of the target object to be tracked.

[0125] In another embodiment, if the position and appearance of the target object to be tracked changes, the tracker can be updated to achieve long-term tracking.

[0126] The advantages of this embodiment are:

[0127] The method includes a detection process and a tracking process;

[0128] (1) Detection process: obtain the image taken by the aircraft, recorded as the first image; correct the first image through the motion compensation algorithm to obtain the corrected first image, recorded as the second image; detect the target object to be tracked in the first image and the second image. Map the pixel position detected in the second image to the first image for position verification again, so as to obtain the final result image with the detection target frame, recorded as the third image;

[0129] (2) Tracking process: The tracker is trained by detecting pixel features of the target object to be tracked in the third image, and the tracker is used to track the target object to be tracked in the image captured by the arbitrary aircraft.

[0130] This embodiment performs motion compensation on the first image to eliminate image differences caused by the linear change law of the device moving with the aircraft, reduce interference, and improve detection accuracy; then, according to the linear change law corresponding to the pixel points in the continuous image calculated above and the flight trajectory of the aircraft, the second detection frame of the multiple second pixel positions extracted from the second image is mapped to the first image, and the similarity between the first detection frame and the second detection frame of the multiple first pixel positions extracted from the first image is calculated; then, the first detection frame or the second detection frame above the target threshold of the similarity is retained, and the first detection frame and the second detection frame below the target threshold of the similarity are smoothed to obtain a third detection frame, and then the third detection frame is mapped to the first image to obtain a third image, so as to improve the accuracy of detecting the target object to be tracked from the third image, and finally, the preset tracker is trained using the third image to improve the tracking accuracy of the trained tracker on the target object to be tracked in the image taken by the aircraft.

[0131] like Figure 2 and Figure 3 One embodiment of the present application provides a target tracking device, comprising:

[0132] Aircraft 100 and ground terminal 200;

[0133] The aircraft 100 mainly realizes high-speed flight. The ground terminal 200 receives data transmitted by the aircraft and transmits data to the aircraft, and displays the flight attitude, flight data, target detection and other conditions of the aircraft to the operator.

[0134] Aircraft 100 includes:

[0135] The main control module 1000, the communication module 2000, the image acquisition module 3000, and the power module 4000. The communication module 2000 is mainly connected to the ground terminal 200; the image acquisition module 3000 is mainly used to acquire the first image; and the power module 4000 is mainly used to supply power.

[0136] The main control module includes:

[0137] The image acquisition unit 1100 is used to acquire a first image; the first image is an image taken by the aircraft at a first moment by a corresponding shooting device during flight;

[0138] A motion compensation unit 1200 is used to perform motion compensation on the first image according to a linear change rule of pixel points in at least two consecutive frames of images captured by the shooting device corresponding to the flight trajectory of the aircraft, so as to obtain a second image;

[0139] The first detection unit 1300 is used to detect a plurality of first pixel positions of a target object to be tracked in a first image and a first detection frame corresponding to the plurality of first pixel positions;

[0140] The second detection unit 1400 is used to detect a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0141] An image mapping unit 1500 is used to map a plurality of second pixel positions and a second detection frame in the second image into the first image according to a linear variation rule;

[0142] A similarity calculation unit 1600 is used to calculate the similarity between the first detection frame and the second detection frame in the first image;

[0143] The image generating unit 1700 is configured to draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame that is smoothed based on the first detection frame and the second detection frame when the degree of similarity is less than or equal to the target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the degree of similarity is greater than the target threshold;

[0144] The object tracking unit 1800 is used to train a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the shooting device at the second moment according to the trained tracker, wherein the second moment is after the first moment.

[0145] It should be noted that since the aircraft-based target detection and tracking device in this embodiment is based on the same inventive concept as the above-mentioned aircraft-based target detection and tracking method, the corresponding contents in the embodiment of the aircraft-based target detection and tracking method are also applicable to the aircraft-based target detection and tracking device embodiment, and will not be described in detail here.

[0146] like Figure 4 The present application also provides an electronic device, which includes:

[0147] at least one memory;

[0148] at least one processor;

[0149] at least one program;

[0150] The programs are stored in the memory, and the processor executes at least one program to implement the above-mentioned aircraft-based target detection and tracking method implemented in the present disclosure.

[0151] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0152] The electronic device according to the embodiment of the present application is described in detail below.

[0153] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0154] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the program code is stored in the memory 1700, and the processor 1600 calls and executes the target detection and tracking method based on an aircraft in the embodiment of the present invention.

[0155] Input / output interface 1800, used to implement information input and output;

[0156] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0157] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0158] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0159] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned aircraft-based target detection and tracking method.

[0160] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0162] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0165] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0166] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0171] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A target detection and tracking method based on an aircraft, characterized in that: The method comprises: Acquire a first image; the first image is an image taken by the aircraft at a first moment by a corresponding photographing device during flight; According to the linear variation law of pixel points in at least two consecutive frames of images captured by the shooting device corresponding to the flight trajectory of the aircraft, motion compensation is performed on the first image to obtain a second image; Detecting a plurality of first pixel positions of a target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions; Detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions; According to the linear variation rule, mapping the plurality of second pixel positions and the second detection frame in the second image into the first image; Calculating a similarity between the first detection frame and the second detection frame in the first image; Draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame that is smoothed based on the first detection frame and the second detection frame when the similarity is less than or equal to the target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity is greater than the target threshold; A preset tracker is trained according to the third image to track the target object to be tracked in the image captured by the shooting device at a second moment according to the trained tracker, wherein the second moment is after the first moment.

2. The target detection and tracking method based on an aircraft according to claim 1, characterized in that: Calculating the similarity between the first detection frame and the second detection frame in the first image captured at the first moment includes: Calculate a first central pixel in the first detection frame in the first image; Calculate a second central pixel in the second detection frame in the first image; determining a Euclidean distance between the first central pixel and the second central pixel; The similarity between the first detection frame and the second detection frame is calculated according to the Euclidean distance.

3. The target detection and tracking method based on an aircraft according to claim 2, characterized in that: The first detection frame and the second detection frame are smoothed by the following formula to obtain a third detection frame: Where s is the target threshold, is the first detection frame, is the second detection frame, is the third detection frame.

4. The method for detecting and tracking an aircraft-based target according to claim 1, characterized in that: The step of training a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the shooting device at the second moment according to the trained tracker, comprises: Extracting pixel features of the target object to be tracked in the third image; Training a preset tracker according to the pixel features to obtain a trained tracker; the tracker is a classifier; Determining the position of the target object to be tracked in the image captured by the shooting device at the second moment according to the trained tracker; wherein the trained tracker determines the position of the target object to be tracked in the image captured by the shooting device at the second moment, comprising: Performing Gaussian kernel function dot multiplication on the pixel features in the trained tracker to obtain a response map output by the trained tracker; The position with the largest response value is found in the response map, and the maximum position is used as the detected position of the target object to be tracked in the image captured by the shooting device at the second moment.

5. The method for detecting and tracking an aircraft-based target according to claim 4, characterized in that: The trained tracker performs Gaussian kernel function dot multiplication on the pixel features in the frequency domain.

6. The method for detecting and tracking a target based on an aircraft according to claim 1, characterized in that: The performing motion compensation on the first image according to the linear change rule of pixel points in at least two consecutive frames of images captured by the shooting device corresponding to the flight trajectory of the aircraft includes: Acquire at least two frames of continuous images captured by the shooting device; Selecting a pixel point to be tracked in a first frame of the at least two consecutive frames of images; Determine the position of the pixel point to be tracked in the images of other frames of the at least two consecutive images except the first frame; Determining the motion trajectory of the pixel to be tracked according to the position; According to the motion trajectory, a global motion model is constructed to characterize the linear change law of pixel points corresponding to the flight trajectory of the aircraft; Motion compensation is performed on the first image according to the global motion model.

7. The method for detecting and tracking a target based on an aircraft according to claim 1, characterized in that: The detecting a plurality of first pixel positions of the target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions includes: Detecting a plurality of first pixel positions of the target object to be tracked in the first image by using a target detection model obtained based on neural network training, and detecting a first detection frame corresponding to the plurality of first pixel positions; The detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions includes: The target detection model is used to detect multiple second pixel positions of the target object to be tracked in the second image, and a second detection frame corresponding to the multiple second pixel positions is detected.

8. A target detection and tracking device based on an aircraft, characterized in that: The device comprises: An image acquisition unit, used to acquire a first image; the first image is an image taken by the aircraft at a first moment during flight by a corresponding shooting device; a motion compensation unit, configured to perform motion compensation on the first image according to a linear change rule of pixel points in at least two consecutive frames of images captured by the capturing device corresponding to the flight trajectory of the aircraft, so as to obtain a second image; A first detection unit, configured to detect a plurality of first pixel positions of the target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions; A second detection unit, configured to detect a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions; An image mapping unit, configured to map the plurality of second pixel positions and the second detection frame in the second image into the first image according to the linear variation rule; a similarity calculation unit, configured to calculate a similarity between the first detection frame and the second detection frame in the first image; an image generating unit, configured to draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame smoothed according to the first detection frame and the second detection frame when the similarity is less than or equal to a target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity is greater than a target threshold; An object tracking unit is used to train a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the shooting device at a second moment according to the trained tracker, wherein the second moment is after the first moment.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the aircraft-based target detection and tracking method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the aircraft-based target detection and tracking method according to any one of claims 1 to 7.

Images