An aircraft-based target detection and tracking method, device, equipment and medium

Abstract

The application relates to an aircraft-based target detection and tracking method, device, equipment and medium. The method first acquires a first image of an aircraft, then performs motion compensation to obtain a second image; a plurality of first pixel positions are extracted from the first image and a plurality of second pixel positions are extracted from the second image; a second detection frame of the plurality of second pixel positions in the second image is mapped in the first image; a first detection frame or a second detection frame with a similarity between the first detection frame and the second detection frame higher than a target threshold is retained; the first detection frame and the second detection frame with a similarity lower than the target threshold are smoothed to obtain a third image converted from the first image, so as to improve the accuracy of detecting a target object to be tracked from the third image; and finally, a preset tracker is trained by using the third image, so as to improve the tracking accuracy of the trained tracker on the target object to be tracked in the image of the aircraft.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of target detection, and in particular to a target detection and tracking method and device based on an aircraft, an apparatus, and a medium. BACKGROUND

[0002] Autonomous tracking of targets by an aircraft is a research direction of great importance to researchers.

[0003] Currently, target detection and tracking of high-speed aircrafts mostly use high-definition cameras mounted thereon, and the camera images are transmitted to a ground terminal in real time for target identification, target confirmation, target locking, and target tracking by an operator. This tracking and detection process is cumbersome and involves many manual intervention steps, which can easily result in missed targets. SUMMARY

[0004] The following is a summary of the subject matter of the detailed description herein. This summary is not intended to limit the scope of the claims.

[0005] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a target detection and tracking method and device based on an aircraft, which can improve the accuracy of detecting a target object to be tracked from an image, thereby achieving more accurate target tracking.

[0006] A first aspect of embodiments of the present application provides a target detection and tracking method based on an aircraft, the method comprising:

[0007] obtaining a first image, the first image being an image captured by a corresponding imaging device of the aircraft at a first time during flight;

[0008] performing motion compensation on the first image according to a linear variation law of pixel points in at least two consecutive images captured by the imaging device corresponding to a flight trajectory of the aircraft, to obtain a second image;

[0009] detecting a plurality of first pixel positions of a target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions;

[0010] detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0011] mapping the plurality of second pixel positions and the second detection frame in the second image to the first image according to the linear variation law;

[0012] calculating a similarity degree between the first detection frame and the second detection frame in the first image;

[0013] drawing a third detection frame in the first image, to obtain a third image, wherein the third detection frame is a detection frame obtained by smoothing the first detection frame and the second detection frame when the similarity degree is less than or equal to a target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity degree is greater than the target threshold

[0014] The embodiment has at least the following beneficial effects:

[0015] The embodiment can eliminate image differences caused by linear variation of the device with the aircraft motion, reduce interference, and improve detection accuracy by performing motion compensation on the first image. Then, the second detection frame of the plurality of second pixel positions extracted from the second image is mapped in the first image according to the linear variation of the pixel points in the continuous images corresponding to the flight trajectory of the aircraft, the similarity degree between the first detection frame of the plurality of first pixel positions extracted from the first image and the second detection frame is calculated, then the first detection frame or the second detection frame higher than the target threshold of the similarity degree is retained, and the first detection frame and the second detection frame lower than the target threshold of the similarity degree are smoothed to obtain a third detection frame, and then the third detection frame is mapped in the first image to obtain a third image, so as to improve the accuracy of detecting the target object to be tracked from the third image, and finally the third image is used to train the preset tracker to improve the tracking accuracy of the trained tracker to the target object to be tracked in the image shot by the aircraft.

[0016] The detection and tracking of the target object to be tracked can improve the accuracy of detecting and tracking the target object to be tracked.

[0017] In some embodiments of the present application, the similarity degree between the first detection frame and the second detection frame in the first image shot at the first time is calculated, including:

[0018] calculating a first center pixel in the first detection frame in the first image;

[0019] calculating a second center pixel in the second detection frame in the first image;

[0020] determining the Euclidean distance between the first center pixel and the second center pixel;

[0021] calculating the similarity degree between the first detection frame and the second detection frame according to the Euclidean distance.

[0022] In some embodiments of the present application, the first detection frame and the second detection frame are smoothed to obtain a third detection frame by the following formula:

[0023]

[0024] wherein s is a target threshold value, is a first detection frame, is a second detection frame, is a third detection frame.

[0025] In some embodiments of the present application, the training of the preset tracker according to the third image includes:

[0026] extracting pixel features in the target object to be tracked in the third image;

[0027] training the preset tracker according to the pixel features to obtain a trained tracker; the tracker is a classifier;

[0028] determining the position of the target object to be tracked in the image captured by the shooting device at the second time according to the trained tracker; wherein the determination of the position of the target object to be tracked in the image captured by the shooting device at the second time according to the trained tracker includes:

[0029] performing Gaussian kernel function point multiplication on the pixel features in the trained tracker to obtain a response map output by the trained tracker;

[0030] finding the position with the maximum response value in the response map, and taking the position as the detected position of the target object to be tracked in the image captured by the shooting device at the second time.

[0031] In some embodiments of the present application, the trained tracker performs Gaussian kernel function point multiplication on the pixel features in the frequency domain.

[0032] In some embodiments of the present application, the motion compensation of the first image according to the linear variation rule corresponding to the flight trajectory of the pixel points in the at least two continuous images captured by the shooting device includes:

[0033] obtaining at least two continuous images captured by the shooting device;

[0034] selecting a pixel point to be tracked in a first image among the at least two continuous images;

[0035] determining the position of the pixel point to be tracked in images other than the first image among the at least two continuous images;

[0036] determining the motion trajectory of the pixel point to be tracked according to the position.

[0037] According to the motion trajectory, a global motion model is constructed to represent a linear change rule of a pixel point corresponding to the flight trajectory of the aircraft.

[0038] According to the global motion model, motion compensation is performed on the first image.

[0039] In some embodiments of the present application, the detection of the plurality of first pixel positions of the target object to be tracked in the first image and the first detection frame corresponding to the plurality of first pixel positions comprises:

[0040] The plurality of first pixel positions of the target object to be tracked in the first image are detected, and the first detection frame corresponding to the plurality of first pixel positions is detected, by a target detection model trained based on a neural network.

[0041] The detection of the plurality of second pixel positions of the target object to be tracked in the second image and the second detection frame corresponding to the plurality of second pixel positions comprises:

[0042] The plurality of second pixel positions of the target object to be tracked in the second image are detected, and the second detection frame corresponding to the plurality of second pixel positions is detected, by the target detection model.

[0043] A second aspect of the embodiments of the present application provides a target detection and tracking device based on an aircraft, which comprises:

[0044] An image acquisition unit is configured to acquire a first image; the first image is an image captured by a corresponding shooting device at a first time during the flight of an aircraft;

[0045] A motion compensation unit is configured to perform motion compensation on the first image according to a linear change rule of a pixel point in at least two continuous images captured by the shooting device corresponding to the flight trajectory of the aircraft, to obtain a second image;

[0046] A first detection unit is configured to detect a plurality of first pixel positions of a target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions;

[0047] A second detection unit is configured to detect a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0048] An image mapping unit is configured to map the plurality of second pixel positions and the second detection frame in the second image in the first image according to the linear change rule.

[0049] a similarity calculation unit configured to calculate a similarity degree between the first bounding box and the second bounding box in the first image;

[0050] an image generation unit configured to draw a third bounding box in the first image to obtain a third image, wherein the third bounding box is a bounding box smoothed according to the first bounding box and the second bounding box when the similarity degree is less than or equal to a target threshold, or is a bounding box composed of the first bounding box or the second bounding box when the similarity degree is greater than the target threshold;

[0051] an object tracking unit configured to train a preset tracker according to the third image, and to track the target object in an image captured by the shooting device at a second time according to the trained tracker, wherein the second time is after the first time.

[0052] The third aspect of the embodiment of the present application provides an electronic device, including at least one control processor and a memory in communication connection with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the target detection and tracking method based on the flying object.

[0053] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions for enabling a computer to execute the target detection and tracking method based on the flying object.

[0054] It can be understood that the beneficial effects of the second aspect to the fourth aspect and the related technology are the same as the beneficial effects of the first aspect and the related technology, and reference can be made to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or related technical description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 is a flowchart of an embodiment of the target detection and tracking method based on the flying object provided by the present application;

[0057] Figure 2is a structural schematic diagram of an embodiment of the target tracking device provided by the present application;

[0058] Figure 3 is Figure 2 is a structural schematic diagram of an embodiment of the target detection and tracking device based on the aircraft provided by the master module in

[0059] Figure 4 is a structural schematic diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0061] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0063] As Figure 1 , one embodiment of the present application provides a target detection and tracking method based on an aircraft, the method comprising the following steps:

[0064] Step S110, acquiring a first image; the first image is an image captured by the corresponding shooting device of the aircraft during the flight at a first time;

[0065] Step S120, performing motion compensation on the first image according to the linear variation law of the pixel points in at least two consecutive images captured by the shooting device corresponding to the flight trajectory of the aircraft, to obtain a second image;

[0066] Step S130, detecting a plurality of first pixel positions of a target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions;

[0067] Step S140, detecting a plurality of second pixel positions of a target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0068] Step S150, mapping the plurality of second pixel positions and the second detection frame in the second image to the first image according to a linear change rule;

[0069] Step S160, calculating a similarity degree between the first detection frame and the second detection frame in the first image;

[0070] Step S170, drawing a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame smoothed according to the first detection frame and the second detection frame when the similarity degree is less than or equal to a target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity degree is greater than the target threshold;

[0071] Step S180, training a preset tracker according to the third image, so as to track the target object to be tracked in the image captured by the shooting device at the second time according to the trained tracker, wherein the second time is after the first time.

[0072] In step S110, the aircraft can be a drone, an airplane, a rocket, an airship, etc. The shooting device is arranged on the aircraft, which can be a camera or a video camera, which is not limited here. The shooting device can capture images during the flight of the aircraft. Here, the image captured by the aircraft at the first time is referred to as the first image.

[0073] It should be noted that the number of first images is not limited here, which can be one frame of image or multiple frames of images.

[0074] In step S120, the first image captured due to the high-speed flight of the aircraft has distortion, which is not conducive to target detection and is prone to missed detection and false detection. According to experimental analysis, since the shooting device moves with the aircraft during the flight of the aircraft, most of the pixels of the image captured by the shooting device will follow the motion pattern of the aircraft. The motion pattern of the overall movement of such a scene can be described by constructing a global motion model. By estimating this global motion model, the image difference caused by the motion can be predicted, and the first image can be compensated to obtain a second image.

[0075] Therefore, step S120 includes the following steps:

[0076] At least two consecutive images captured by the shooting device are obtained. Here, the consecutive images refer to consecutive image frames, for example, the shooting device captures images every certain period of time to obtain consecutive images, or the shooting device captures videos and samples consecutive images from the videos by a sampling algorithm.

[0077] A pixel point to be tracked is selected in the first frame of the consecutive images.

[0078] In some embodiments, the pixel points to be tracked are corner points and edge points. Whether a pixel point is a corner point is determined based on the gray level change caused by moving the local window of the image in any direction. If a pixel point has a very large gray level change in the window no matter in which direction the window is moved, it is considered as a corner point. The calculation principle is shown as follows:

[0079] R = det(M) - kgtrace 2 (M) (1)

[0080] wherein R is the response function value of the corner point; det(M) is the determinant of M; trace(M) is the trace of M; k is a hyper parameter, generally taking a value between 0.04 and 0.06; wherein the expressions of det(M) and trace(M) are as follows:

[0081]

[0082] wherein I is the gradient value in x and y directions; w(x, y) is the Gaussian weight of each pixel in the window;

[0083]

[0084] which represents the sum of the elements on the diagonal of the matrix.

[0085] The positions of the pixel points to be tracked in the other images except the first image in the continuous images are determined. The corner points and edge points can be tracked to determine their motion trajectories.

[0086] According to the positions, the motion trajectories of the pixel points to be tracked are determined; and according to the motion trajectories of the corner points and edge points, a global motion model containing translation and affine change is constructed.

[0087] According to the motion trajectories, a global motion model representing the linear change rule of the pixel points corresponding to the flight trajectory of the aircraft is constructed.

[0088] According to the global motion model, the motion compensation of the first image is performed.

[0089] The affine change is the change of the pixel points in the space of the changed image, i.e. the linear change of the two-dimensional coordinates to the two-dimensional coordinates, and the calculation process is as follows:

[0090]

[0091] wherein (x', y') is the coordinate after transformation; (x, y) is the original coordinate; a-f are six degrees of freedom, a, b, d, e are four parameters controlling the linear transformation, and c, f control the translation bias.

[0092] The first image is motion compensated by using the constructed global motion model, and the motion compensated first image is encoded and transmitted to an H.265 / HEVC decoder for decoding and reconstructing the first image to obtain the second image.

[0093] In step S130, a target detection model trained based on a neural network can be used to detect the target object to be tracked in the first image. The target object to be tracked is an object that needs to be tracked. The target object to be tracked can be specified in advance, for example, a drone in front of the aircraft.

[0094] For example, according to the artificial neural network, the shallow appearance features and deep semantic features of the target object to be tracked are learned in advance, the model weight related to the feature information of the target object to be tracked is obtained, and then the model weight is used for feature extraction, feature information filtering, feature attribute discrimination, non-maximum suppression and other technical means on the entire first image to identify and detect the target object to be tracked. The confidence of the detected target object to be tracked is compared with the set threshold. If the confidence is higher than the set threshold, the detection is successful, and the position, label, confidence, detection box and other information of the detected target object to be tracked are output.

[0095] In step S140, the target detection model can also be used to detect the target object to be tracked in the second image, which will not be described in detail.

[0096] In step S150, the relevant pixels in the second image can be mapped to the first image according to the inverse operation of the global motion model given above.

[0097] It should be noted that the first image includes the first pixel position in the original first image and the corresponding first detection box, and the second pixel position in the original second image and the corresponding second detection box.

[0098] In step S160, the similarity between the first detection box and the second detection box in the first image is calculated, including the following steps:

[0099] The first center pixel in the first detection box in the first image is calculated;

[0100] The second center pixel in the second detection box in the first image is calculated;

[0101] The Euclidean distance between the first center pixel and the second center pixel is determined;

[0102] The similarity between the first detection box and the second detection box is calculated according to the Euclidean distance.

[0103] In step S170, a third detection frame is drawn in the first image to obtain a third image, wherein the third detection frame is a detection frame obtained by smoothing the first detection frame and the second detection frame when the similarity degree is less than or equal to the target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity degree is greater than the target threshold.

[0104] Specifically, the third detection frame is obtained by smoothing the first detection frame and the second detection frame according to the following formula:

[0105]

[0106] wherein s is the target threshold, is the first detection frame, is the second detection frame, is the third detection frame.

[0107] The present application includes a detection process and a tracking process, wherein the detection process includes the foregoing steps, and the tracking process includes step S180.

[0108] In step S180, the step specifically includes:

[0109] extracting pixel features in the target object to be tracked in the third image;

[0110] training a preset tracker according to the pixel features to obtain a trained tracker; the tracker is a classifier;

[0111] determining the position of the target object to be tracked in the image captured by the shooting device at the second time according to the trained tracker; wherein the determination of the position of the target object to be tracked in the image captured by the shooting device at the second time according to the trained tracker includes:

[0112] performing Gaussian kernel function point multiplication on the pixel features in the trained tracker to obtain a response map output by the trained tracker;

[0113] finding the position with the maximum response value in the response map, and taking the position with the maximum response value as the position of the detected target object to be tracked in the image captured by the shooting device at the second time.

[0114] In the present embodiment, the target object to be tracked is confirmed by the ground operator on the third image, for example, the third image is first transmitted to the ground, and the target object to be tracked is confirmed by the ground operator, and then the information of the target object to be tracked transmitted by the ground is received to extract the pixel features of the target object to be tracked. It can also be that the ground confirms in advance, that is, the ground operator confirms the target object to be tracked in the first image, and then receives the information of the target object to be tracked transmitted by the ground. The pixel features herein include but are not limited to gradient histogram and color histogram.

[0115] The embodiment trains a discriminant classifier as a tracker to realize tracking of the target object to be tracked, and uses a circulant matrix and fast Fourier transform to accelerate the calculation process, so as to ensure tracking accuracy and improve tracking speed. The specific process is as follows:

[0116] First, the training process of the tracker is introduced.

[0117] The pixel features of the sample image (i.e. multiple frames of the third image) are converted to the frequency domain by using fast Fourier transform to accelerate subsequent calculation, and the response map output by the classifier is obtained by point multiplication of the pixel features of the target object to be tracked and the Gaussian kernel function. Then, the response map is used to train the tracker to obtain the weight coefficients of the tracker.

[0118] The response calculation of the tracker is as follows:

[0119]

[0120] wherein, is the Fourier transform of the response; is the first row of the Fourier transform of the kernel correlation matrix between x and z, is the coefficient obtained by solving the ridge regression problem, and e is the point multiplication;

[0121] The solution formula of the ridge regression in the frequency domain is as follows:

[0122]

[0123] wherein, is the Fourier transform of w, x, and y; x is the training sample matrix; y is the sample label; and λ is the regularization parameter;

[0124] In the use process, for any image collected by the aircraft, the pixel features of the target object to be tracked are also extracted therefrom, the pixel features are subjected to point multiplication operation of the Gaussian kernel function with the trained tracker to obtain a response map, and the position of the maximum response in the response map is found. The position is the position of the target object to be tracked in the image. Further, the tracking of the target object to be tracked is realized.

[0125] In another embodiment, the position and appearance of the target object to be tracked change, and the tracker can be updated to realize long-time tracking.

[0126] The advantages of the embodiment are as follows:

[0127] The method includes a detection process and a tracking process.

[0128] (1) Detection process: obtain the image taken by the aircraft, denoted as a first image; and correct the first image by a motion compensation algorithm to obtain a corrected first image, denoted as a second image; detect the target object to be tracked in the first image and the second image. Map the pixel position detected in the second image to the first image for position verification again, thereby obtaining a result image with a detection target frame, denoted as a third image;

[0129] (2) Tracking process: train the tracker by detecting the pixel features of the target object to be tracked in the third image, and use the tracker to track the target object to be tracked in the image collected by the aircraft.

[0130] The embodiment can eliminate the image difference caused by the linear change rule of the device moving with the aircraft, reduce interference, and improve detection accuracy by motion compensation on the first image. Then, the second detection frame of the plurality of second pixel positions extracted from the second image is mapped in the first image according to the linear change rule of the pixel points in the continuous image corresponding to the flight trajectory of the aircraft, the similarity between the first detection frame of the plurality of first pixel positions extracted from the first image and the second detection frame is calculated, then the first detection frame or the second detection frame higher than the target threshold of the similarity degree is retained, and the first detection frame and the second detection frame lower than the target threshold of the similarity degree are smoothed to obtain a third detection frame, and then the third detection frame is mapped in the first image to obtain a third image, so as to improve the accuracy of detecting the target object to be tracked from the third image, and finally train the preset tracker by using the third image, so as to improve the tracking accuracy of the trained tracker on the target object to be tracked in the image taken by the aircraft.

[0131] As Figure 2 and Figure 3 , one embodiment of the present application provides a target tracking device, comprising:

[0132] an aircraft 100 and a ground terminal 200;

[0133] The aircraft 100 mainly realizes high-speed flight. The ground terminal 200 receives data transmitted by the aircraft and transmits data to the aircraft, and displays the flight attitude, flight data, target detection, etc. of the aircraft to the operator.

[0134] The aircraft 100 comprises:

[0135] a main control module 1000, a communication module 2000, an image acquisition module 3000, and a power supply module 4000. The communication module 2000 is mainly connected with the ground terminal 200; the image acquisition module 3000 is mainly used for acquiring a first image; and the power supply module 4000 is mainly used for power supply.

[0136] The main control module includes:

[0137] The image acquisition unit 1100 is configured to acquire a first image; the first image is an image captured by a corresponding photographing device at a first moment during the flight of the aircraft;

[0138] A motion compensation unit 1200 is configured to perform motion compensation on a first image according to a linear variation law of pixels in at least two consecutive frames of images captured by a capturing device along a flight trajectory of the aircraft, thereby obtaining a second image.

[0139] The first detection unit 1300 is configured to detect a plurality of first pixel positions of a target object to be tracked in a first image and a first detection frame corresponding to the plurality of first pixel positions;

[0140] The second detection unit 1400 is configured to detect a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions;

[0141] An image mapping unit 1500 is configured to map a plurality of second pixel positions and a second detection frame in the second image to the first image according to a linear variation rule;

[0142] A similarity calculation unit 1600 is configured to calculate a similarity between a first detection frame and a second detection frame in a first image;

[0143] Image generation unit 1700 is configured to draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame obtained by smoothing the first detection frame and the second detection frame when the degree of similarity is less than or equal to a target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the degree of similarity is greater than the target threshold;

[0144] The object tracking unit 1800 is used to train a preset tracker based on the third image, so as to track the target object to be tracked in the image captured by the shooting device at the second moment according to the trained tracker, wherein the second moment is after the first moment.

[0145] It should be noted that since the aircraft-based target detection and tracking device in this embodiment is based on the same inventive concept as the above-mentioned aircraft-based target detection and tracking method, the corresponding contents in the embodiment of the aircraft-based target detection and tracking method are also applicable to the embodiment of the aircraft-based target detection and tracking device, and will not be described in detail here.

[0146] like Figure 4 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0147] at least one memory;

[0148] at least one processor;

[0149] at least one program;

[0150] The program is stored in the memory, and the processor executes the at least one program to implement the aircraft-based target detection and tracking method of the above-mentioned embodiments of the present disclosure.

[0151] The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, and the like.

[0152] The electronic device of the embodiments of the present application is described in detail below.

[0153] The processor 1600 can be implemented in a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0154] The memory 1700 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the program codes are saved in the memory 1700 and are called and executed by the processor 1600 to implement the aircraft-based target detection and tracking method of the embodiments of the present application.

[0155] The input / output interface 1800 is used to realize information input and output.

[0156] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0157] The bus 2000 transmits information between various components (such as the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900) of the device.

[0158] The processor 1600, the memory 1700, the input / output interface 1800 and the communication interface 1900 are communicatively connected with each other within the device through the bus 2000.

[0159] The embodiment of the present application further provides a storage medium, which is a computer readable storage medium, and stores computer executable instructions for causing a computer to execute the aircraft-based target detection and tracking method.

[0160] The memory is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0161] The embodiments described in the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0162] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0163] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0164] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0165] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0166] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0167] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment of the present application.

[0169] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0170] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or in other words, the part of the prior art that contributes to the present application, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0171] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and variations.

Claims

1. An aircraft-based target detection and tracking method, characterized in that, The method comprises: acquiring a first image; the first image is an image taken by a corresponding shooting device at a first time during a flight of an aircraft; performing motion compensation on the first image according to a linear variation law of pixel points in at least two frames of continuous images taken by the shooting device corresponding to a flight trajectory of the aircraft, to obtain a second image; detecting a plurality of first pixel positions of a target object to be tracked in the first image and a first detection frame corresponding to the plurality of first pixel positions; detecting a plurality of second pixel positions of the target object to be tracked in the second image and a second detection frame corresponding to the plurality of second pixel positions; mapping the plurality of second pixel positions and the second detection frame in the second image to the first image according to the linear variation law; calculating a similarity degree between the first detection frame and the second detection frame in the first image; drawing a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame smoothed according to the first detection frame and the second detection frame when the similarity degree is less than or equal to a target threshold, or is a detection frame composed of the first detection frame or the second detection frame when the similarity degree is greater than the target threshold; training a preset tracker according to the third image to track the target object to be tracked in an image taken by the shooting device at a second time according to the trained tracker, wherein the second time is after the first time; the training of the preset tracker according to the third image to track the target object to be tracked in the image taken by the shooting device at the second time according to the trained tracker comprises: extracting pixel features in the target object to be tracked in the third image; training a preset tracker according to the pixel features to obtain a trained tracker; the tracker is a classifier; determining a position of the target object to be tracked in the image taken by the shooting device at the second time according to the trained tracker; wherein the determination of the position of the target object to be tracked in the image taken by the shooting device at the second time according to the trained tracker comprises: performing Gaussian kernel function point multiplication on the pixel features in the trained tracker to obtain a response map output by the trained tracker; finding a position with the maximum response value in the response map, and taking the position as the detected position of the target object to be tracked in the image taken by the shooting device at the second time.

2. The aircraft-based target detection and tracking method of claim 1, wherein, calculating the similarity degree between the first detection frame and the second detection frame in the first image taken at the first time comprises: calculating a first center pixel in the first detection frame in the first image; calculating a second center pixel in the second detection frame in the first image; determining an Euclidean distance between the first center pixel and the second center pixel; The similarity between the first bounding box and the second bounding box is calculated according to the Euclidean distance.

3. The aircraft-based target detection and tracking method of claim 2, wherein, The first bounding box and the second bounding box are smoothed by the following formula to obtain a third bounding box: wherein, is a target threshold value, is a first detection frame, is a second detection frame, is a third detection frame.

4. The aircraft-based target detection and tracking method of claim 1, wherein, The trained tracker performs point multiplication of the pixel features in the frequency domain with a Gaussian kernel function.

5. The aircraft-based target detection and tracking method of claim 1, wherein, The motion compensation on the first image includes: Obtaining at least two continuous images captured by the shooting device; Selecting a pixel point to be tracked in a first frame image among the at least two continuous images; Determining the position of the pixel point to be tracked in images other than the first frame image among the at least two continuous images; According to the position, determining the motion trajectory of the pixel point to be tracked; According to the motion trajectory, constructing a global motion model representing the linear variation rule of the pixel point corresponding to the flight trajectory of the aircraft; According to the global motion model, performing motion compensation on the first image.

6. The aircraft-based target detection and tracking method of claim 1, wherein, The detection of the plurality of first pixel positions of the target object to be tracked in the first image and the first bounding box corresponding to the plurality of first pixel positions includes: Detecting the plurality of first pixel positions of the target object to be tracked in the first image and the first bounding box corresponding to the plurality of first pixel positions by a target detection model trained based on a neural network; The detection of the plurality of second pixel positions of the target object to be tracked in the second image and the second bounding box corresponding to the plurality of second pixel positions includes: Detecting the plurality of second pixel positions of the target object to be tracked in the second image and the second bounding box corresponding to the plurality of second pixel positions by the target detection model.

7. An aircraft-based target detection and tracking apparatus, characterized by, The device includes: An image acquisition unit configured to acquire a first image; the first image is an image captured by a corresponding shooting device at a first time by an aircraft during flight; A motion compensation unit configured to perform motion compensation on the first image according to a linear variation rule of pixel points in at least two continuous images captured by the shooting device corresponding to the flight trajectory of the aircraft to obtain a second image; A first detection unit configured to detect a plurality of first pixel positions of a target object to be tracked in the first image and a first bounding box corresponding to the plurality of first pixel positions; A second detection unit configured to detect a plurality of second pixel positions of the target object to be tracked in the second image and a second bounding box corresponding to the plurality of second pixel positions; An image mapping unit configured to map the plurality of second pixel positions and the second bounding box in the second image in the first image according to the linear variation rule; A similarity calculation unit configured to calculate the similarity between the first bounding box and the second bounding box in the first image; The image generation unit is configured to draw a third detection frame in the first image to obtain a third image, wherein the third detection frame is a detection frame obtained by smoothing the first detection frame and the second detection frame when the similarity degree is less than or equal to a target threshold, or the third detection frame is a detection frame composed of the first detection frame or the second detection frame when the similarity degree is greater than the target threshold; The object tracking unit is configured to train a preset tracker according to the third image, and track the target object in the image captured by the shooting device at a second time according to the trained tracker, wherein the second time is after the first time. The training of the preset tracker according to the third image, and the tracking of the target object in the image captured by the shooting device at a second time according to the trained tracker, include: extracting pixel features in the target object in the third image; training a preset tracker according to the pixel features to obtain a trained tracker; the tracker is a classifier; determining the position of the target object in the image captured by the shooting device at a second time according to the trained tracker; wherein the determination of the position of the target object in the image captured by the shooting device at a second time according to the trained tracker includes: performing Gaussian kernel function point multiplication on the pixel features in the trained tracker to obtain a response map output by the trained tracker; finding a position with the maximum response value in the response map, and taking the position as the detected position of the target object in the image captured by the shooting device at a second time.

8. An electronic device, comprising: The computer readable storage medium stores computer executable instructions for causing a computer to execute the aerial vehicle based target detection and tracking method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions for causing a computer to execute the aerial vehicle based target detection and tracking method according to any one of claims 1 to 6.

Images