Target tracking method and device, electronic device, and storage medium

Abstract

The present application provides a target tracking method and device, electronic device, and storage medium, the method comprising: obtaining an image frame sequence; obtaining a differential image sequence by calculating the inter-frame difference of the image frame sequence; using the image frame sequence and the differential image sequence as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network; and tracking the target according to the detection result sequence. This solution can improve the accuracy of target tracking.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target tracking method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Tracking of moving objects is usually divided into two main parts: object detection and object tracking.

[0003] Object detection refers to inputting an image or an image with depth and outputting the positions of all objects in the image. Most of the currently better performing methods for object detection are based on convolutional neural networks. Commonly used neural network structures include DenseBox and RetinaNet. These networks directly input the entire image and output the object rectangle and confidence score. Object detection can be completed after simple post-processing (such as non-maximum suppression).

[0004] Object tracking refers to inputting the object detection results of several consecutive frames, outputting the motion trajectories of these objects, and suppressing the false positives in the detection results as much as possible. Traditional target tracking algorithms connect the detection results of adjacent frames through Intersect over union and perform motion estimation based on the principle that the distance between objects in adjacent frames will not be too far. After generating the motion trajectory, the parts that are too short are identified as false positives and removed. The target tracking algorithm based on neural networks inputs the images deducted in the detection stage, outputs its features, and connects the same object in multiple frames by comparing the consistency of the features to derive the motion trajectory.

[0005] Neural network-based object detection does not take advantage of the nature of "moving objects", so it is necessary to detect all objects in the field of view, which requires a large amount of training data and has a high false positive probability. Moving object tracking related methods have limited use of original image information, and most of them are only tracked based on detection results, with low accuracy. Summary of the invention

[0006] The embodiment of the present application provides a target tracking method to improve tracking accuracy.

[0007] The present application provides a target tracking method, including:

[0008] Get image frame sequence;

[0009] By calculating the inter-frame difference of the image frame sequence, a differential image sequence is obtained;

[0010] Using the image frame sequence and the differential image sequence as inputs of a trained object detection network to obtain a detection result sequence output by the object detection network;

[0011] Target tracking is performed according to the detection result sequence.

[0012] In one embodiment, the target tracking according to the detection result sequence includes:

[0013] According to the image frame sequence, the differential image sequence and the detection result sequence, the detection results that do not belong to the target object are removed to obtain a target position sequence;

[0014] According to the target position sequence, a target motion trajectory is obtained.

[0015] In one embodiment, the detection result sequence includes a moving object position sequence and a local image sequence, and removing the detection results that do not belong to the target object according to the image frame sequence, the differential image sequence, and the detection result sequence to obtain the target position sequence includes:

[0016] Using the local image sequence as an input of a trained feature extraction network to obtain an image feature sequence corresponding to the local image sequence output by the feature extraction network;

[0017] According to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence, the moving object positions that do not belong to the target object are removed to obtain a target position sequence.

[0018] In one embodiment, removing the moving object positions that do not belong to the target object according to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence to obtain the target position sequence includes:

[0019] Using the image frame sequence and the differential image sequence as inputs of an optical flow calculation network, and obtaining an optical flow information offset between adjacent frames output by the optical flow calculation network;

[0020] Superimposing the optical flow information offset between the adjacent frames onto the target object position of the previous frame to obtain the predicted target object position of the next frame;

[0021] According to the moving object position sequence and the image feature sequence, the moving object position whose difference between the image feature and the moving object position in the next frame and the image feature and the predicted target object position of the target object is greater than a first threshold is removed to obtain the target object position of the next frame, and the target object position of each frame constitutes the target position sequence.

[0022] In one embodiment, removing the moving object positions that do not belong to the target object according to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence to obtain the target position sequence includes:

[0023] The position and motion speed of the target object in the previous frame are used as inputs of the Kalman filter model to obtain the predicted position of the target object in the next frame output by the Kalman filter model.

[0024] Calculating an intersection-and-union ratio between a predicted position of a target object in a subsequent frame and a position of a moving object in an image of the subsequent frame, and removing a moving object position in the subsequent frame whose intersection-and-union ratio is less than a second threshold;

[0025] According to the moving object position sequence and the image feature sequence, the moving object position whose difference between the image features and the moving object position in the next frame and the image features and the predicted target object position of the target object is greater than a third threshold is removed to obtain the target object position of the next frame, and the target object position of each frame constitutes the target position sequence.

[0026] In one embodiment, the method further comprises:

[0027] Calculating the target average depth according to the target depth information of each frame in the image frame sequence;

[0028] The target object position whose target depth information differs from the target average depth by more than a fourth threshold is removed.

[0029] In one embodiment, the method further comprises:

[0030] Generating a continuous multi-frame target distribution heat map according to the differential image sequence;

[0031] According to the target object position of the specified frame image, calculating the weighted average value of the target object position in the target distribution heat map corresponding to the specified frame image;

[0032] Taking a weighted average of the target object positions in the target distribution heat map corresponding to each frame of the image, a thermal response mean is obtained;

[0033] The target object positions whose thermal response mean is less than the fifth threshold are removed.

[0034] In one embodiment, the step of obtaining a differential image sequence by calculating the inter-frame difference of the image frame sequence includes:

[0035] For each two adjacent frames of images in the image frame sequence, the pixel difference of each pixel point in each channel between the two adjacent frames of images is calculated to obtain differential image information corresponding to each two adjacent frames of images;

[0036] The differential image information corresponding to each two adjacent frames of images is arranged in sequence to obtain the differential image sequence.

[0037] In one embodiment, the step of obtaining a differential image sequence by calculating the inter-frame difference of the image frame sequence includes:

[0038] Taking a designated frame image in the image frame sequence as a reference, calculating a first average image of a first plurality of frame images and a second average image of a second plurality of frame images;

[0039] Calculating the pixel difference between the first mean image and the second mean image to obtain differential image information corresponding to the specified frame image;

[0040] The differential image information corresponding to the specified frame image is sorted in sequence to obtain the differential image sequence.

[0041] In one embodiment, the image frame sequence and the differential image sequence are used as inputs of a trained object detection network to obtain a detection result sequence output by the object detection network, including:

[0042] fusing the image frame sequence with the differential image sequence to obtain a fused image sequence;

[0043] The fused image sequence is used as an input of the trained object detection network to obtain a detection result sequence of the fused image sequence output by the object detection network.

[0044] In one embodiment, the fusion of the image frame sequence and the differential image sequence to obtain a fused image sequence is performed in any one of the following ways:

[0045] Method 1: splicing the image frame sequence and the differential image sequence on channels to obtain the fused image sequence;

[0046] Method 2: multiplying the differential image sequence and the image frame sequence pixel by pixel to obtain an intermediate image sequence;

[0047] splicing the difference image sequence, the image frame sequence and the intermediate image sequence on channels to obtain the fused image sequence;

[0048] Method three, performing a pooling operation on the differential image sequence to obtain a reduced image sequence;

[0049] Enlarging the reduced image sequence to the same size as the image frame sequence to obtain an image sequence of the same scale;

[0050] The same-scale image sequence and the image frame sequence are added pixel by pixel to obtain the fused image sequence.

[0051] The present application also provides a target tracking device, including:

[0052] An image acquisition module, used for acquiring an image frame sequence;

[0053] A difference calculation module, used for obtaining a difference image sequence by calculating the inter-frame difference of the image frame sequence;

[0054] A target detection module, used to use the image frame sequence and the differential image sequence as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network;

[0055] The target tracking module is used to track the target according to the detection result sequence.

[0056] The present application also provides an electronic device, the electronic device comprising:

[0057] processor;

[0058] a memory for storing processor-executable instructions;

[0059] Wherein, the processor is configured to execute the above-mentioned target tracking method.

[0060] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete the above-mentioned target tracking method.

[0061] The technical solution provided by the above-mentioned embodiments of the present application obtains a differential image sequence by calculating the inter-frame differences of an image frame sequence; this method can filter out stationary objects, and all detected objects are in motion, which are suitable for subsequent moving object tracking. The differential image sequence is input into the target detection model together with the original image frame sequence, and a frame-level detection result sequence can be obtained, thereby improving the accuracy of moving object detection, and then tracking the target based on the detection result sequence can suppress the false detection rate and improve the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0063] Figure 1 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0064] Figure 2 It is a flowchart of a target tracking method provided by an embodiment of the present application;

[0065] Figure 3 yes Figure 2 Detailed flowchart of step S220 in the corresponding embodiment;

[0066] Figure 4yes Figure 2 Detailed flowchart of step S230 in the corresponding embodiment;

[0067] Figure 5 yes Figure 2 Detailed flowchart of step S240 in the corresponding embodiment;

[0068] Figure 6 yes Figure 5 Detailed flowchart of step S241 in the corresponding embodiment;

[0069] Figure 7 It is a flowchart of a target tracking method based on an optical flow computing network;

[0070] Figure 8 It is a flowchart of a target tracking method based on a Kalman filter model;

[0071] Fig. 9 yes Figure 7 or Figure 8 A flowchart of a target tracking method provided by an embodiment of the present application based on a corresponding embodiment;

[0072] Fig.10 It is a schematic diagram of the principle architecture of a target tracking method provided by an embodiment of the present application;

[0073] Fig.11 It is a block diagram of a target tracking device shown in one embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0075] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0076] Figure 1 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 can be used to execute the target tracking method provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 storing processor executable instructions. The processor 102 is configured to execute the target tracking method provided in the following embodiment of the present application.

[0077] The processor 102 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.

[0078] The memory 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the target tracking method described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0079] In one embodiment, Figure 1 The electronic device 100 shown may also include an input device 106, an output device 108, and a data acquisition device 110, which are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device 100 may also have other components and structures as required.

[0080] The input device 106 may be a device for a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, etc. The data acquisition device 110 may acquire an image of an object, and store the acquired image in the memory 104 for use by other components. Exemplarily, the data acquisition device 110 may be a camera.

[0081] In one embodiment, the various components in the example electronic device 100 for implementing the target tracking method of the embodiment of the present application can be integrated or dispersed, such as integrating the processor 102, memory 104, input device 106 and output device 108 into one, while separating the data acquisition device 110.

[0082] In one embodiment, the example electronic device 100 for implementing the target tracking method of the embodiment of the present application can be implemented as a smart terminal such as a smart phone, a tablet computer, a smart watch, a vehicle-mounted device, etc.

[0083] Figure 2 is a flow chart of the target tracking method provided in the embodiment of the present application. Figure 2 As shown, the method includes the following steps S210-S250.

[0084] Step S210: Acquire an image frame sequence.

[0085] Among them, the image frame sequence refers to multiple frames of images taken continuously, which can be RGB (red, green, and blue) format images or RGBD (red, green, blue + depth) format images. For example, the image frame sequence can be a continuous multi-frame image of a video. Each frame of the image may contain one or more moving objects. The moving objects in the embodiments of the present application can be moving people, animals, vehicles or other objects.

[0086] Step S220: obtaining a differential image sequence by calculating the inter-frame differences of the image frame sequence.

[0087] The inter-frame difference refers to the pixel difference between each channel (e.g., red, green, blue, depth) of each pixel between image frames, and the absolute value of the inter-frame difference of all pixels constitutes differential image information. Since the image frame sequence includes multiple image frames, the differential image sequence may include multiple differential image information. In one embodiment, a differential image model may be used to input the image frame sequence into the differential image model, and the differential image model outputs a differential image sequence containing one or more differential image information.

[0088] In one embodiment, for every two adjacent frames of images in an image frame sequence, the pixel difference of each pixel point in each channel between the two adjacent frames of images can be calculated to obtain differential image information corresponding to each two adjacent frames of images; then, the differential image information corresponding to each two adjacent frames of images are arranged in sequence to obtain the differential image sequence.

[0089] It should be noted that at the pixel position of a stationary object, the pixel difference is 0. However, at the pixel position of a moving object, the pixel difference is not 0. Therefore, calculating the pixel difference of each pixel in each channel between frames has the ability to detect the position of a moving object in the input image.

[0090] Each two adjacent frames of images may include the first frame and the second frame, the second frame and the third frame, the third frame and the fourth frame, and so on. The differential image information refers to the absolute value of the pixel difference of all channels of all pixels between two adjacent frames of images, and the differential image information has the same size and the same number of channels as the image frame. For example, the absolute value of the pixel difference of each channel (R, G, B, D four channels) of each pixel between the first frame and the second frame is calculated to obtain differential image information 1, and the absolute value of the pixel difference of each channel of each pixel between the second frame and the third frame is calculated to obtain differential image information 2. By analogy, differential image information 1, differential image information 2...all differential image information are arranged in order to form a differential image sequence. As needed, before calculating the pixel difference between each channel of each pixel, each frame of the image can be Gaussian blurred in advance, and then the pixel difference of each channel of each pixel between each two adjacent frames of images is calculated. Such processing can reduce the interference caused by changes in pixel values ​​caused by weather, external light, etc., and improve the accuracy of positioning moving objects.

[0091] In another embodiment, if Figure 3 As shown, the differential image sequence can also be obtained through the following steps.

[0092] Step S221: taking a designated frame image in the image frame sequence as a reference, calculating a first average image of the first several frame images and a second average image of the last several frame images.

[0093] Wherein, assuming that the first several frames are the first three frames, the designated frame image may be each frame image starting from the third frame. Assuming that the first several frame images are the first five frames, the designated frame image may be each frame image starting from the fifth frame, and so on. The first several frame images refer to the several frame images before the designated frame image. The next several frame images refer to the several frame images after the designated frame image.

[0094] Assuming that each frame image in the image frame sequence is numbered in sequence, assuming that the designated frame image is the ath frame, for example, the first several frames of images may be images of the a-1th to ath frames, and the next several frames of images may be images of the ath to a+4th frames. For example, the first several frames of images may be images of the a-5th to a-1th frames, and the next several frames of images may be images of the a+1th to a+5th frames.

[0095] The first mean image refers to an image obtained by calculating the pixel mean of each pixel in the previous several frames of images in sequence and taking the mean as the pixel value of the pixel.

[0096] The second mean image refers to an image obtained by calculating the pixel mean of each pixel in subsequent frames of images and using the mean as the pixel value of the pixel.

[0097] Step S222: Calculate the pixel difference between the first mean image and the second mean image to obtain differential image information corresponding to the specified frame image.

[0098] Specifically, the pixel difference between the first mean image and the second mean image can be calculated for each pixel point in turn, and the absolute value of the pixel difference of all pixels constitutes the differential image information. Assuming that a certain differential image information is based on the pixel difference of the first mean image of several frames of images before the a-th frame and the second mean image of several frames of images after the a-th frame, then this differential image information can be considered as the differential image information corresponding to the a-th frame.

[0099] Step S223: sorting the differential image information corresponding to the specified frame image in order to obtain the differential image sequence.

[0100] For example, starting from the 5th frame, the first mean image of frames 1-5 and the second mean image of frames 6-10 are calculated, and the absolute value of the pixel difference between the first mean image and the second mean image is calculated to obtain differential image information 1; for the 6th frame, the first mean image of the first 2-6 frames and the second mean image of the last 7-11 frames are calculated, and the absolute value of the pixel difference between the first mean image and the second mean image is calculated to obtain differential image information 2; for the 7th frame, the first mean image of the first 3-7 frames and the second mean image of the last 8-12 frames are calculated, and the absolute value of the pixel difference between the first mean image and the second mean image is calculated to obtain differential image information 3, and so on, all differential image information is arranged in order to obtain a differential image sequence.

[0101] Step S230: using the image frame sequence and the differential image sequence as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network.

[0102] The object detection network can be trained using a large number of images with known moving object positions. An image may contain more than one moving object, so the detection result sequence may include the positions of one or more moving objects in each frame.

[0103] In one embodiment, if Figure 4 As shown, the above step S230 specifically includes step S231-step S232.

[0104] Step S231: Fusing the image frame sequence with the differential image sequence to obtain a fused image sequence.

[0105] The fused image sequence includes multiple frames of continuous fused images, and the fused image refers to an image obtained by fusing image frames in the image frame sequence with corresponding differential image information.

[0106] In one embodiment, assuming that the differential image information is differential image information between the ath frame and the a+1th frame image, the ath frame may be fused with the differential image information to obtain a fused image corresponding to the ath frame.

[0107] In another embodiment, assuming that the differential image information is the differential image information between the a-1th frame and the ath frame image, the ath frame may be fused with the differential image information to obtain a fused image corresponding to the ath frame.

[0108] In other embodiments, assuming that the differential image information is the pixel difference between the first average image of the first several frames before the a-th frame and the second average image of the second several frames after the a-th frame, the a-th frame can be fused with the differential image information to obtain a fused image of the a-th frame. Therefore, the fused image corresponding to each frame image in the image sequence constitutes a fused image sequence.

[0109] Among them, the fusion method can be one of the following three methods.

[0110] Method 1: splicing the image frame sequence and the differential image sequence on channels to obtain the fused image sequence.

[0111] For example, the specified frame image and the differential image information corresponding to the specified frame image may be spliced ​​on the channel to obtain a fused image corresponding to the specified frame image. The fused images corresponding to each frame image constitute a fused image sequence.

[0112] Assume that the specified frame image is four-channel. Since the differential image information directly calculates the difference of each pixel, it is also four-channel. Stitching on the channels means stitching to form eight channels, that is, each pixel in the fused image contains R, G, B, and D values, as well as R difference, G difference, B difference, and D difference.

[0113] Method 2: multiplying the difference image sequence and the image frame sequence pixel by pixel to obtain an intermediate image sequence; splicing the difference image sequence, the image frame sequence and the intermediate image sequence on the channel to obtain the fused image sequence;

[0114] For example, the differential image information corresponding to the specified frame image can be multiplied pixel by pixel with the specified frame image to obtain an intermediate image corresponding to the specified frame image; the specified frame image, the differential image information corresponding to the specified frame image, and the intermediate image are spliced ​​on the channel to obtain a fused image corresponding to the specified frame image, so that the fused images corresponding to each frame image constitute a fused image sequence.

[0115] Since the inter-frame difference may be negative, the absolute value of the inter-frame difference can be taken first to obtain the differential image information, and then multiplied pixel by pixel with the specified frame image. The image obtained in this way can be called the intermediate image. Assuming that the specified frame image is four-channel, the differential image information is also four-channel, and the intermediate image is also four-channel. Therefore, the specified frame image, the differential image information and the intermediate image can be spliced ​​on the channel to obtain a fused image with original 4 channels + differential 4 channels + dot product 4 channels = 12 channels for each pixel. This method can extract areas with higher differential responses from the specified frame image, allowing the target detection network to better detect the target position.

[0116] Method three: performing a pooling operation on the differential image sequence to obtain a reduced image sequence; enlarging the reduced image sequence to the same size as the image frame sequence to obtain a same-scale image sequence; performing a pixel-by-pixel addition operation on the same-scale image sequence and the image frame sequence to obtain the fused image sequence.

[0117] For example, a pooling operation is performed on the differential image information corresponding to the specified frame image to obtain a reduced-size image. The reduced-size image sequence includes the reduced-size image of the differential image information of each frame image. The reduced-size image is enlarged to the same size as the specified frame image to obtain a same-scale image; the same-scale image sequence includes the same-scale image corresponding to each reduced-size image; the same-scale image is added pixel by pixel to the specified frame image to obtain a fused image corresponding to the specified frame image. The fused images corresponding to each frame image constitute a fused image sequence.

[0118] Among them, the pooling operation is an operation that does not change the number of image channels but reduces the image size. Taking the commonly used Max-Pooling as an example, assuming the parameter is 2×2, and a 1920×1080×4 image is input, then the output is 960×540×4. In simple terms, each channel is calculated separately, and a maximum value (Max) is taken for each 2×2 pixel area as the result. This operation will make the image smaller by a certain proportion. Therefore, the image obtained after the differential image information is pooled can be called a reduced-size image.

[0119] In order to superimpose with the specified frame image, the reduced-size image needs to be subsequently enlarged to the same size as the specified frame image. For distinction, this is called the same-scale image. Afterwards, for each pixel point, the pixel value of the pixel point in the same-scale image can be added to the pixel value of the pixel point in the specified frame image to obtain the pixel value of the pixel point in the fused image. In this way, the fused image can still be 4-channel, and the moving object information can be added to the specified frame image without increasing the amount of calculation, so that the target detection network is more inclined to moving object detection.

[0120] Step S232: using the fused image sequence as input of the trained object detection network to obtain a detection result sequence of the fused image sequence output by the object detection network.

[0121] The target detection network can be a convolutional neural network model with a pre-trained RetinaNet structure. With the fused image as input, the target detection network can output a moving object rectangular frame and confidence. After non-maximum suppression, a credible moving object rectangular frame position can be obtained. The detection result sequence may include the rectangular frame position of the moving object detected from each frame of the fused image, that is, a moving object position sequence. In one embodiment, the rectangular frame position in the fused image can also be cropped to obtain a local image of the corresponding running object, and each frame obtains a local image sequence from the cropped local image. The local image is input into the feature extraction network, and the image features of the local image can be extracted to obtain an image feature sequence. The image features are used to characterize whether they belong to the same target (for example, the same car, the same person), and the image features of the same target are similar in different images.

[0122] Step S240: performing target tracking according to the detection result sequence.

[0123] The detection result sequence may include the moving position of the running object in each frame. Target tracking refers to connecting the positions of the target at different times to form the target motion trajectory. Assuming there is only one moving object, the line connecting the moving positions of the moving object in each frame is the target motion trajectory. Assuming there are multiple moving objects, the positions of moving objects that do not belong to the same target can be filtered out based on the moving object positions and features, and the remaining lines connecting the moving object positions in each frame are used as the target motion trajectory.

[0124] The following is a detailed description of the process of filtering out the positions of moving objects that do not belong to the same target.

[0125] In one embodiment, if Figure 5 As shown, the above step S240 specifically includes: step S241: removing the detection results that do not belong to the target object according to the image frame sequence, the differential image sequence and the detection result sequence to obtain a target position sequence; step S242: obtaining the target motion trajectory according to the target position sequence.

[0126] The target object can be any one of multiple moving objects, such as a vehicle. The target position sequence refers to the position of the target object in each frame of the image. The line connecting the positions of the target object in each frame of the image constitutes the target motion trajectory.

[0127] In one embodiment, the detection result sequence may include the moving position of the running object in each frame, and removing the detection results that do not belong to the target object means removing the position of the moving object that does not belong to the target object. For example, based on the position of the target object in the first frame image, the position of the target object in the subsequent frames can be predicted, and the moving objects whose positions in the subsequent frames are significantly different from the predicted positions can be removed; based on the image features of the target object in the first frame image, the moving objects whose image features in the subsequent frames are significantly different from those of the target object can be removed, so in one embodiment, a feature extraction network can also be used to extract image features.

[0128] In one embodiment, the detection result sequence may include a moving object position sequence and a local image sequence. The local image refers to an image where the moving object is located, and the image where the moving object is located in each frame of image constitutes the local image sequence.

[0129] like Figure 6 As shown, the above step S241 specifically includes steps S2411 to S2422.

[0130] Step S2411: using the local image sequence as input of a trained feature extraction network to obtain an image feature sequence corresponding to the local image sequence output by the feature extraction network.

[0131] The image feature sequence refers to a sequence of image features of each local image. Image features are used to represent the features of a moving object. Different moving objects have different features, so that the image features can be used to distinguish whether they belong to the same moving object. A local image can be an image of the area where the moving object is located captured from the original image frame based on the position of the moving object indicated by the detection result, or an image of the area where each moving object is located captured from the fused image.

[0132] In one embodiment, siamese network reasoning can be used for feature extraction, and the loss function during training is designed as feature difference, which can be calculated using L1 distance (l1-distance) or L2 distance (l2-distance). The feature difference extracted after different data augmentation (Data Augmentation) of the same input can be used as the positive sample loss function, so that it can extract the same or similar features from the positive sample as much as possible, and the opposite of the feature difference extracted from different inputs can be used as the negative sample loss function, so that it can extract different features from the negative sample as much as possible. The neural network trained in this way tends to extract the same features from the information of the same object in different images.

[0133] In other embodiments, the labeled video can be used to extract different frames of the same object as positive sample inputs to train the twin network. Compared with data enhancement, this method is more inclined to extract the same features of the same object in the same video. In another embodiment, the twin network can be pre-trained by using input with differential image information as training data, which can improve the ability to extract frame differential information during reasoning.

[0134] Step S2422: According to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence, the moving object positions that do not belong to the target object are removed to obtain a target position sequence.

[0135] Image frame sequences, differential image sequences, moving object position sequences, and image feature sequences can be used as inputs to the tracking algorithm, and outputs the target motion trajectory. Tracking algorithms can be divided into two categories: using neural networks (such as the optical flow calculation network described below) and not using neural networks (such as using the Kalman filter model described below). The following describes these two methods in detail.

[0136] In one embodiment, if Figure 7 As shown, the solution using the optical flow calculation network includes the following steps S710 to S730.

[0137] Step S710: using the image frame sequence and the differential image sequence as inputs of an optical flow calculation network, and obtaining an optical flow information offset between adjacent frames output by the optical flow calculation network.

[0138] The optical flow calculation network can be a convolutional neural network with a FlowNet 2.0 structure. The input is two images, a fused image of the previous frame and a fused image of the next frame. The fused image is obtained based on the image frame sequence and the differential image sequence, which has been described in detail above and will not be repeated here. The image frame can be an RGBD image, where D represents depth information, so the fused image can contain the depth information and differential image information of the image frame, thereby improving the accuracy of optical flow calculation. The output of the optical flow calculation network is called the optical flow information offset, which is used to characterize the offset of each pixel in the two images (increment in the x direction and increment in the y direction).

[0139] Step S720: superimpose the optical flow information offset between adjacent frames onto the target object position of the previous frame to obtain the predicted target object position of the next frame.

[0140] Specifically, the optical flow information offset is added to the target object position of the previous frame to obtain the predicted target object position of the next frame. In one embodiment, before the addition, weighted averaging or Gaussian smoothing and other operations can be performed within the target object position range to improve the accuracy, and then the target object position of the previous frame is added to the optical flow information offset to obtain the estimated predicted target object position of the next frame. The predicted target object position refers to the estimated value of the position of the target object in the next frame. The target object position of the previous frame refers to the detection result of the target object in the previous frame, which can be obtained through the target detection network, and the target object can be any moving object. The previous frame and the next frame are relative, and can be the previous frame and the next frame in two adjacent image frames.

[0141] Step S730: According to the moving object position sequence and the image feature sequence, remove the moving object position whose difference between the image features and the moving object position in the next frame and the image features and the target object predicted position is greater than a first threshold, and obtain the target object position of the next frame. The target object position of each frame constitutes the target position sequence.

[0142] The above differences may include feature differences and position differences. The moving object positions whose difference d1 between the image features in the next frame and the image features of the target object is greater than a threshold a are removed, and the moving object positions whose difference d2 between the moving object position and the predicted position of the target object in the next frame is greater than a threshold b are removed. The above differences d1 and d2 may be calculated using an L1 distance paradigm or an L2 distance paradigm. In one embodiment, since the position information is a rectangular frame, the intersection over union (IoU) may be further calculated as an evaluation index. If the intersection over union between the moving object position and the predicted position of the target object in the next frame is greater than a threshold c, it may be considered that the motion estimation is satisfied. On the contrary, the moving object positions whose intersection over union is less than a threshold c in the next frame are removed. The moving object positions that are finally left in the next frame may be considered as the target object positions in the next frame. By analogy, the target object positions of each frame are obtained, and a target position sequence is obtained. The motion trajectory of the target object (i.e., the target motion trajectory) may be obtained by connecting the target object positions of each frame.

[0143] In one embodiment, if Figure 8 As shown, the solution using the Kalman filter model includes the following steps S810 to S830.

[0144] Step S810: taking the position and motion speed of the target object in the previous frame as inputs of the Kalman filter model, and obtaining the predicted position of the target object in the next frame output by the Kalman filter model.

[0145] Kalman filtering is a method for first-order linear estimation of multiple variables, where the variables can be the position and speed of the target object. When the historical changes of these variables are known (in this task, the motion trajectory of the previous frames), Kalman filtering can provide future predicted values ​​of these variables (in this task, the predicted position and predicted speed of the next frame). In one embodiment, the motion trajectory of the previous frames can also be obtained using the above steps S710-S730, and then the position and speed of the target object of the next frame are predicted based on the position and speed of the target object of the last frame in the trajectory of the previous frames. Among them, the speed of movement can be obtained by dividing the movement distance by the movement time.

[0146] Step S820: Calculate the intersection-and-union ratio between the target object prediction position in the next frame and the moving object position in the next frame image, and remove the moving object position in the next frame whose intersection-and-union ratio is less than a second threshold.

[0147] Since the position of the target object can be represented in the form of a rectangular frame. Assuming that there are a positions of moving objects in the previous frame and b positions of moving objects in the next frame, then theoretically there may be a×b different motion trajectories. Therefore, some completely non-intersecting results can be removed in advance to reduce the amount of calculation. Specifically, after estimating the predicted position of target A in the next frame based on the position of target A in the previous frame, the position detection results of moving objects in the next frame whose intersection-and-union ratio is less than the second threshold can be removed by calculating the intersection-and-union ratio between the predicted position of target A in the next frame and the positions of all moving objects in the next frame. For distinction, the threshold corresponding to the intersection-and-union ratio is called the second threshold.

[0148] Step S830: Based on the moving object position sequence and the image feature sequence, remove the moving object position whose difference between the image features and the moving object position in the next frame and the image features and the target object predicted position is greater than a third threshold, and obtain the target object position of the next frame. The target object position of each frame constitutes the target position sequence.

[0149] The above step S830 can be implemented with reference to step S730, which will not be described in detail here.

[0150] Since the distance between the same target and the camera will not change too much in a short period of time, in order to improve the accuracy of target tracking, in one embodiment, the target average depth can be further calculated based on the target depth information of each frame image in the image frame sequence; and the target object position whose difference between the target depth information and the target average depth exceeds a fourth threshold is removed.

[0151] It should be noted that the image frame can be in RGBD format, that is, the moving object in each frame of the image has depth information in addition to color information. The target depth information can be the depth information of a certain target (such as a car), and the depth information is used to characterize the distance between the target and the camera. If the difference between the target depth information of the target object in a certain frame and the target average depth of the target object is large, for example, greater than the fourth threshold, it can be considered that the target object position of the current frame is incorrect, and the target object position of the current frame can be removed, thereby further suppressing false positives and improving tracking accuracy.

[0152] To further improve the tracking accuracy and suppress false positives, such as Fig. 9 As shown, the following steps S910 to S940 may be further performed.

[0153] Step S910: generating a continuous multi-frame target distribution heat map according to the differential image sequence.

[0154] In one embodiment, the inter-frame difference can be removed from the position where the inter-frame difference of the depth information exceeds the threshold, thereby reducing the difference caused by the switching of the foreground and the background. At this time, the remaining pixels have a greater ratio of containing moving objects, so they can be used as a distribution heat map of moving objects. This operation can be performed on each frame of differential image information in the differential image sequence in turn, and the inter-frame difference is removed from the position where the inter-frame difference of the depth information exceeds the threshold. The inter-frame difference at other positions is taken as an absolute value or calculated by a nonlinear function 1 / (|x|+1) to obtain the values ​​of other positions. The image thus obtained can be called a target distribution heat map. In the target distribution heat map, the grayscale value of the pixel where the moving object is located is greater than 0, while the grayscale value of other pixels is 0.

[0155] Step S920: According to the target object position of the designated frame image, a weighted average value of the target object position in the target distribution heat map corresponding to the designated frame image is calculated.

[0156] For example, assuming that the target object position of target A in the ath frame is obtained based on the above calculation, the target distribution heat map of this differential image information can be obtained according to the differential image information between the ath frame and the a+1th frame. Or the target distribution heat map of this differential image information can be obtained according to the differential image information between the a-1th frame and the ath frame. Different target distribution heat maps can be obtained by different calculation methods of differential image information. Then, the weighted average of the grayscale values ​​of all pixels in the target object position area (the target object position can be represented by a rectangular box) in the target distribution heat map is calculated. By analogy, for the target object position in each frame of the image, a weighted average value can be calculated accordingly.

[0157] Step S930: performing weighted averaging on the weighted average values ​​of the target object positions in the target distribution heat map corresponding to each frame of image to obtain a thermal response mean.

[0158] For example, according to the target A, the weighted average value x1 is calculated in the first frame image, the weighted average value x2 is calculated in the second frame image, the weighted average value x3 is calculated in the third frame image... x1, x2, x3... can be weighted averaged, and the obtained value is called the thermal response mean.

[0159] Step S940: remove the target object positions whose thermal response mean is less than the fifth threshold.

[0160] For example, if the mean thermal response of target A is less than the fifth threshold, the target object position of target A in each frame image can be removed, that is, the motion trajectory of target A is deleted, thereby removing trajectories with too low responses and improving tracking accuracy.

[0161] Fig.10 FIG. 1 is a schematic diagram of the principle architecture of a target tracking method provided by an embodiment of the present application. Fig.10 As shown, the following steps are included:

[0162] Step S1001: A video in RGB or RGBD format is used as input of a frame difference model, and a difference image sequence is output. The calculation process of the frame difference model is shown in step S220.

[0163] Step S1002: The still image sequence (i.e., image frame sequence) and the differential image sequence are used as inputs of a dual-input detection convolutional network (i.e., target detection network), and the object position and local image are output. For details, see step S230.

[0164] Step S1003: Use the local image as input of the feature extraction network to obtain the object position and feature vector. The feature extraction method is as described above.

[0165] Step S1004: The still image sequence, the differential image sequence, the object position and the feature vector are used as inputs of a three-input tracking algorithm, and the tracking trajectory of the moving object is output. The three-input tracking algorithm can refer to the optical flow calculation network and Kalman filter model mentioned above.

[0166] The above-mentioned embodiments of the present application can obtain video input with depth through video acquisition and radar equipment, obtain a differential image sequence through frame difference calculation, and input it into the convolutional neural network model for reasoning together with the original input, and obtain the frame-level moving object position respectively. After multiple filtering conditions such as intersection and union ratio calculation, position difference, feature difference, depth difference, distribution heat map density, etc., the accurate moving object trajectory can be obtained, the false detection rate can be suppressed, and the accuracy of the moving object detection system can be improved.

[0167] The following is an embodiment of the device of the present application, which can be used to execute the above-mentioned target tracking method embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the target tracking method of the present application.

[0168] Fig.11 FIG. 1 is a block diagram of a target tracking device according to an embodiment of the present application. Fig.11 As shown, the device includes: an image acquisition module 1110, a difference calculation module 1120, a target detection module 1130 and a target tracking module 1140.

[0169] An image acquisition module 1110 is used to acquire an image frame sequence;

[0170] The difference calculation module 1120 is used to obtain a difference image sequence by calculating the inter-frame difference of the image frame sequence;

[0171] The target detection module 1130 is used to use the image frame sequence and the differential image sequence as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network;

[0172] The target tracking module 1140 is used to track the target according to the detection result sequence.

[0173] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned target tracking method, which will not be repeated here.

[0174] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0176] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

Claims

1. A target tracking method, characterized in that: include: Get image frame sequence; By calculating the inter-frame difference of the image frame sequence, a differential image sequence is obtained; Using the image frame sequence and the differential image sequence as inputs of a trained object detection network to obtain a detection result sequence output by the object detection network; Performing target tracking according to the detection result sequence; The image frame sequence and the differential image sequence are used as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network, including: fusing the image frame sequence with the differential image sequence to obtain a fused image sequence; The fused image sequence is used as an input of the trained object detection network to obtain a detection result sequence of the fused image sequence output by the object detection network.

2. The method according to claim 1, characterized in that The performing target tracking according to the detection result sequence includes: According to the image frame sequence, the differential image sequence and the detection result sequence, the detection results that do not belong to the target object are removed to obtain a target position sequence; According to the target position sequence, a target motion trajectory is obtained.

3. The method according to claim 2, characterized in that The detection result sequence includes a moving object position sequence and a local image sequence, and the detection results that do not belong to the target object are removed according to the image frame sequence, the differential image sequence and the detection result sequence to obtain the target position sequence, including: Using the local image sequence as an input of a trained feature extraction network to obtain an image feature sequence corresponding to the local image sequence output by the feature extraction network; According to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence, the moving object positions that do not belong to the target object are removed to obtain a target position sequence.

4. The method according to claim 3, characterized in that: The removing the moving object positions that do not belong to the target object according to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence to obtain the target position sequence includes: Using the image frame sequence and the differential image sequence as inputs of an optical flow calculation network, and obtaining an optical flow information offset between adjacent frames output by the optical flow calculation network; Superimposing the optical flow information offset between the adjacent frames onto the target object position of the previous frame to obtain the predicted target object position of the next frame; According to the moving object position sequence and the image feature sequence, the moving object position whose difference between the image feature and the moving object position in the next frame and the image feature and the predicted target object position of the target object is greater than a first threshold is removed to obtain the target object position of the next frame, and the target object position of each frame constitutes the target position sequence.

5. The method according to claim 3, characterized in that: The removing the moving object positions that do not belong to the target object according to the image frame sequence, the differential image sequence, the moving object position sequence and the image feature sequence to obtain the target position sequence includes: The position and motion speed of the target object in the previous frame are used as inputs of the Kalman filter model to obtain the predicted position of the target object in the next frame output by the Kalman filter model; Calculating an intersection-and-union ratio between a predicted position of a target object in a subsequent frame and a position of a moving object in an image of the subsequent frame, and removing a moving object position in the subsequent frame whose intersection-and-union ratio is less than a second threshold; According to the moving object position sequence and the image feature sequence, the moving object position whose difference between the image features and the moving object position in the next frame and the image features and the predicted target object position of the target object is greater than a third threshold is removed to obtain the target object position of the next frame, and the target object position of each frame constitutes the target position sequence.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Calculating the target average depth according to the target depth information of each frame in the image frame sequence; The target object position whose target depth information differs from the target average depth by more than a fourth threshold is removed.

7. The method according to claim 4 or 5, characterized in that: Also includes: Generating a continuous multi-frame target distribution heat map according to the differential image sequence; According to the target object position of the specified frame image, calculating the weighted average value of the target object position in the target distribution heat map corresponding to the specified frame image; Taking a weighted average of the target object positions in the target distribution heat map corresponding to each frame of the image, a thermal response mean is obtained; The target object positions whose thermal response mean is less than the fifth threshold are removed.

8. The method according to claim 1, characterized in that The step of obtaining a differential image sequence by calculating the inter-frame difference of the image frame sequence comprises: For each two adjacent frames of images in the image frame sequence, the pixel difference of each pixel point in each channel between the two adjacent frames of images is calculated to obtain differential image information corresponding to each two adjacent frames of images; The differential image information corresponding to each two adjacent frames of images is arranged in sequence to obtain the differential image sequence.

9. The method according to claim 1, characterized in that: The step of obtaining a differential image sequence by calculating the inter-frame difference of the image frame sequence comprises: Taking a designated frame image in the image frame sequence as a reference, calculating a first average image of a first plurality of frame images and a second average image of a second plurality of frame images; Calculating the pixel difference between the first mean image and the second mean image to obtain differential image information corresponding to the specified frame image; The differential image information corresponding to the specified frame image is sorted in sequence to obtain the differential image sequence.

10. The method according to claim 1, characterized in that The fusion of the image frame sequence and the differential image sequence to obtain a fused image sequence is performed in any of the following ways: Method 1: splicing the image frame sequence and the differential image sequence on channels to obtain the fused image sequence; Method 2: multiplying the differential image sequence and the image frame sequence pixel by pixel to obtain an intermediate image sequence; splicing the difference image sequence, the image frame sequence and the intermediate image sequence on channels to obtain the fused image sequence; Method three, performing a pooling operation on the differential image sequence to obtain a reduced image sequence; Enlarging the reduced image sequence to the same size as the image frame sequence to obtain an image sequence of the same scale; The same-scale image sequence and the image frame sequence are added pixel by pixel to obtain the fused image sequence.

11. A target tracking device, characterized in that: include: An image acquisition module, used for acquiring an image frame sequence; A difference calculation module, used for obtaining a difference image sequence by calculating the inter-frame difference of the image frame sequence; A target detection module, used to use the image frame sequence and the differential image sequence as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network; A target tracking module, used for tracking the target according to the detection result sequence; The image frame sequence and the differential image sequence are used as inputs of a trained target detection network to obtain a detection result sequence output by the target detection network, including: fusing the image frame sequence with the differential image sequence to obtain a fused image sequence; The fused image sequence is used as an input of the trained object detection network to obtain a detection result sequence of the fused image sequence output by the object detection network.

12. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the target tracking method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the target tracking method described in any one of claims 1-10.

Images