Target tracking method, electronic equipment and related products

Abstract

The embodiment of the present application discloses a target tracking method, an electronic device and related products, which are applied to an electronic device, wherein the electronic device includes a target tracking system, and the target tracking system includes a detection network and a tracking network. The method includes: obtaining a heat map of the i-th frame image and the i-1-th frame image in a video sequence, inputting the heat map of the i-th frame image and the i-1-th frame image into the detection network, obtaining a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2; determining target information and a target tracking trajectory based on the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image; inputting the target information and the target tracking trajectory into the tracking network, obtaining an optimal allocation matrix, and the numerical value in the optimal allocation matrix represents the tracking target to which the target belongs. The use of the embodiment of the present application can improve the target tracking accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target tracking method, electronic equipment and related products. Background Art

[0002] In the prior art, visual target tracking refers to the detection, extraction, identification and tracking of moving targets in image sequences, obtaining the motion parameters of moving targets, such as position, speed, acceleration and motion trajectory, so as to carry out the next step of processing and analysis, realize the understanding of the behavior of moving targets, and complete a higher level of detection tasks. According to the number of tracking targets, the tracking algorithm can be divided into single target tracking and multi-target tracking. Compared with single target tracking, the multi-target tracking problem is more complex and difficult. The multi-target tracking problem needs to consider the position, size and other data of multiple independent targets in the video sequence. The changes in the appearance of multiple targets, different motion modes, the influence of dynamic lighting, and the mutual occlusion, merging and separation between multiple targets are all difficult points in the multi-target tracking problem. Therefore, the problem of how to improve the tracking accuracy of multi-target tracking needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a target tracking method and related products, which can improve target tracking efficiency.

[0004] In a first aspect, an embodiment of the present application provides a target tracking method, which is applied to an electronic device, wherein the electronic device includes a target tracking system, and the target tracking system includes a detection network and a tracking network. The method includes:

[0005] Obtain a heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2;

[0006] Determine target information and target tracking trajectory according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image;

[0007] The target information and the target tracking trajectory are input into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0008] In a second aspect, an embodiment of the present application provides a target tracking device, which is applied to an electronic device, wherein the electronic device includes a target tracking system, the target tracking system includes a detection network and a tracking network, and the device includes: a detection unit, a determination unit and a tracking unit, wherein:

[0009] The detection unit is used to obtain the heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2;

[0010] The determining unit is used to determine the target information and the target tracking trajectory according to the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image;

[0011] The tracking unit is used to input the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, and the numerical values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in the first aspect of the embodiment of the present application.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] The implementation of the embodiments of the present application has the following beneficial effects:

[0016] It can be seen that the target tracking method, electronic device and related products described in the embodiments of the present application are applied to electronic devices, and the electronic device includes a target tracking system, and the target tracking system includes a detection network and a tracking network. The heat map of the i-th frame image and the i-1-th frame image in the video sequence is obtained, and the heat map of the i-th frame image and the i-1-th frame image is input into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2, and the target information and the target tracking trajectory are determined according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, and the target information and the target tracking trajectory are input into the tracking network to obtain the optimal allocation matrix, and the numerical value in the optimal allocation matrix represents the tracking target to which the target belongs. The detection network can realize the confidence feature map, position feature map, position increment feature map and reid feature map based on the heat map of the previous frame and the current frame image. The detection network can accurately detect the target and has a re-identification function, and can accurately grasp the target trajectory and target information, thereby realizing accurate classification of the target, which helps to improve the tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1A It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0019] Figure 1B It is a flowchart of a target tracking method provided by an embodiment of the present application;

[0020] Figure 1C is a demonstration schematic diagram of a detection network provided in an embodiment of the present application;

[0021] Figure 1D is a demonstration schematic diagram of a tracking network provided in an embodiment of the present application;

[0022] Figure 2 is a flowchart of another target tracking method provided in an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0024] Figure 4 It is a block diagram of the functional units of a target tracking device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but in a possible example also includes steps or units that are not listed, or in a possible example also includes other steps or units inherent to these processes, methods, products or devices.

[0026] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The electronic devices involved in the embodiments of the present application may be handheld devices including various image processing functions, intelligent robots, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem, as well as various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. The electronic devices may also be servers or smart home devices.

[0029] In the embodiments of the present application, the smart home device may be at least one of the following: a refrigerator, a washing machine, an electric rice cooker, smart curtains, smart lights, a smart bed, a smart trash can, a microwave oven, an oven, a steamer, an air conditioner, a range hood, a server, a smart door, a smart window, a window door wardrobe, a smart speaker, a smart home, a smart chair, a smart clothes rack, a smart shower, a water dispenser, a water purifier, an air purifier, a doorbell, a monitoring system, a smart garage, a television, a projector, a smart dining table, a smart sofa, a massage chair, a treadmill, etc., and of course, other devices may also be included.

[0030] like Figure 1A As shown, Figure 1A 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a processor, a memory, a signal processor, a transceiver, a display screen, a speaker, a microphone, a random access memory (RAM), a camera, a sensor, and a network module, etc. Among them, the memory, the signal processor DSP, the speaker, the microphone, the RAM, the camera, the sensor, and the network module are connected to the processor, and the transceiver is connected to the signal processor.

[0031] Among them, the processor is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory and calling data stored in the memory, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor can be a central processing unit (CPU), a graphics processing unit (GPU) or a network processor (NPU).

[0032] Furthermore, the processor may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor.

[0033] The memory is used to store software programs and / or modules, and the processor executes various functional applications and target tracking of the electronic device by running the software programs and / or modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a software program required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0034] The sensor includes at least one of the following: a light sensor, a gyroscope, an infrared proximity sensor, a vibration detection sensor, a pressure sensor, and the like. The light sensor, also known as an ambient light sensor, is used to detect the brightness of ambient light. The light sensor may include a photosensitive element and an analog-to-digital converter. The photosensitive element is used to convert the collected light signal into an electrical signal, and the analog-to-digital converter is used to convert the electrical signal into a digital signal. Optionally, the light sensor may further include a signal amplifier, which may amplify the electrical signal converted by the photosensitive element and output it to the analog-to-digital converter. The photosensitive element may include at least one of a photodiode, a phototransistor, a photoresistor, and a silicon photocell.

[0035] The camera may be a visible light camera (a normal viewing angle camera, a wide-angle camera), an infrared camera, or a dual camera (with a ranging function), which is not limited here.

[0036] The network module may be at least one of the following: a Bluetooth module, wireless fidelity (Wi-Fi), etc., which is not limited here.

[0037] Based on the above Figure 1A The electronic device described includes a target tracking system, and the target tracking system includes a detection network and a tracking network, and can execute the following target tracking method, and the specific steps are as follows:

[0038] Obtain a heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2;

[0039] Determine target information and target tracking trajectory according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image;

[0040] The target information and the target tracking trajectory are input into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0041] It can be seen that the electronic device described in the embodiment of the present application includes a target tracking system, which includes a detection network and a tracking network. The heat map of the i-th frame image and the i-1-th frame image in the video sequence is obtained, and the heat map of the i-th frame image and the i-1-th frame image is input into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2. The target information and the target tracking trajectory are determined according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, and the target information and the target tracking trajectory are input into the tracking network to obtain the optimal allocation matrix. The numerical values ​​in the optimal allocation matrix represent the tracking target to which the target belongs. The detection network can realize the confidence feature map, position feature map, position increment feature map and reid feature map based on the heat map of the previous frame and the current frame image. The detection network can accurately detect the target and has a re-identification function, and can accurately grasp the target trajectory and target information, thereby realizing accurate classification of the target, which helps to improve the tracking accuracy.

[0042] See also Figure 1B , Figure 1B is a flow chart of a target tracking method provided by an embodiment of the present application, as shown in the figure, which is applied to Figure 1A The electronic device shown includes a target tracking system, the target tracking system includes a detection network and a tracking network, and the target tracking method includes:

[0043] 101. Obtain heat maps of the i-th frame image and the i-1-th frame image in the video sequence, input the heat maps of the i-th frame image and the i-1-th frame image into the detection network, and obtain a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2.

[0044] Among them, in an embodiment of the present application, the video sequence can be a video image taken by one or more cameras. The video sequence can be a continuous image within a period of time. The electronic device may include a target tracking system, which includes a detection network and a tracking network, the detection network is used for target detection and trajectory detection, and the tracking network is used for target identification and target classification based on the results of target detection and trajectory tracking. The output layer of the detection network adds a corresponding target re-identification feature vector layer and a previous and next frame target position increment layer, merges the detection network, the target re-identification network, and the tracking network into one network, and improves the detection capability of the detection network. In an embodiment of the present application, the target can be at least one of the following: a face, a human body, a license plate, a license plate or other moving objects, etc., which are not limited here. The re-id feature vector can also be called a re-id feature vector map.

[0045] In a specific implementation, the electronic device can obtain the heat map of the i-th frame image and the i-1-th frame image in the video sequence, and input the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2.

[0046] Optionally, the above step 101, inputting the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, may include the following steps:

[0047] 11. The i-th frame image is processed successively through a convolution layer, a BN layer and an activation layer, and the first intermediate result is input into the backbone network for feature extraction. The extracted features are used to respectively determine the confidence feature map, the position feature map and the reid feature vector of the i-th frame image. The confidence feature map of the i-th frame image is also used to optimize the position feature map, and the position feature map is also used to optimize the reid feature vector. The reid feature vector is obtained by a CosFace loss function, and the map of the reid feature vector is a random value between 0 and 1.

[0048] 12. Process the heat map of the i-1th frame image through a convolution layer, a BN layer, and an activation layer in succession to obtain a second intermediate result, perform a convolution operation on the second intermediate result and the extracted features, and determine the position increment feature map based on the result of the convolution operation. The position feature map in turn optimizes the position increment feature map during the back propagation process. The position increment feature map is obtained by the Softmax loss function, and the maps of the position increments are initialized to all 0 during training.

[0049] In the specific implementation, Figure 1CAs shown, the electronic device can process the i-th frame image through a convolution layer, a normalization (Batchnorm, BN) layer, and an activation layer (Sigmoid) in sequence, input the first intermediate result into the backbone network for feature extraction, and use the extracted features to determine the confidence feature map, position feature map, and reid feature vector of the i-th frame image, and then process the heat map of the i-1-th frame image through a convolution layer, a BN layer, and an activation layer in sequence to obtain a second intermediate result, perform a convolution operation on the second intermediate result and the extracted features, and determine the position increment feature map based on the result of the convolution operation. Person Re-ID Re-identification is also called pedestrian re-identification, or ReID for short. When training the incremental feature map output, you can perform data enhancement on the training data, simulate various motion phenomena, and initialize the target incremental map to all 1. Add a branch to the backbone network layer, connect a convolutional layer, output the feature value, and use the Softmax loss function to optimize the incremental feature map. When training the re-id feature vector, initialize the target re-id to all 1, add a branch after the backbone network layer, connect a convolutional layer, output the feature value, and use the CosFace loss function to optimize the re-id Feature vector. The confidence feature map of the i-th frame image is also used to optimize the position feature map, and the position feature map is also used to optimize the reid feature vector, which is equivalent to using the confidence feature map of the i-th frame image to guide the generation of the position feature map, which helps to improve the accuracy of the position feature map. Then, optimizing the reid feature vector through the position feature map can improve the reid generation accuracy, which helps to improve the subsequent target tracking accuracy. The position feature map in turn optimizes the position increment feature map during the back propagation process, so that the position feature map and the position increment feature map represent mutual enhancement, which can improve the accuracy of both and help improve the target capture accuracy.

[0050] In a specific implementation, when the first frame image of a video sequence is input, its corresponding heat map can be initialized to 0 and input into the detection network, and the target information and confidence feature map can be output. The target information can be at least one of the following: target position, target pixel value, target type, etc., which are not limited here. According to the target information, a heat map is rendered. Furthermore, when the second frame image and subsequent images are input, the current frame image and the previous frame target heat map are input, and the current frame image is subjected to convolution, BN, and activation layer operations, and then the features are extracted through the backbone network. The extracted features are subjected to convolution layer, BN layer, and activation layer operations, and the confidence map feature map, position feature map, and reid feature vector map are output respectively; the previous frame target heat map is convolved with the information output by the convolution, BN layer, and backbone network, and then subjected to convolution, BN layer, and activation layer operations to obtain a position increment feature map. The above backbone network can be a residual network or a DLA network.

[0051] Optionally, in the above step 11, after the first intermediate result is input into the backbone network for feature extraction, and the confidence feature map, the position feature map and the reid feature vector of the i-th frame image are respectively determined using the extracted features, the following steps may also be included:

[0052] S1, extracting global features from the feature map after feature extraction to obtain global features;

[0053] S2, performing local area detection on the feature map to obtain a region of interest;

[0054] S3, dividing the global features into S×S grids, where S is an integer greater than 1;

[0055] S4, projecting the region of interest to the S×S grids, extracting a local feature vector of each projected grid using local average pooling to obtain local features; the local features in turn optimize the global feature representation during back propagation;

[0056] S5. Aggregate the global features and the local features to obtain extracted features.

[0057] In a specific implementation, the electronic device can perform global feature extraction on the feature map after feature extraction to obtain global features, and perform local feature extraction on the feature map to obtain local features, and then aggregate the global features and local features to obtain query image features.

[0058] Specifically, the entire feature extraction network can be composed of two modules, one is the global feature extraction module, which is used to extract the most significant global representation, and the other is the local feature extraction module, which is used to extract the detailed features of a specific area. Among them, the local features are derived from the global features. In the back propagation process, the local features will in turn optimize the global feature representation, so that the global features and local area feature representations enhance each other. Finally, the extracted global features and local features are aggregated to obtain the final global features and local area features.

[0059] Furthermore, the feature map can be subjected to local area detection to obtain a specific region of interest ROI of the target (taking humans as an example, eyebrows, lips, nose bridge, etc.), and then the generated ROI is projected to the global feature. Suppose the spatial size of the global feature is S×S, where S is an integer greater than 1. The feature map is divided into S×S grids, and each grid cell overlapping with the ROI will be marked as the part corresponding to the ROI. Then local average pooling is used to extract the local feature vector of each ROI projection area to obtain the local feature.

[0060] Optionally, the electronic device may perform global feature extraction on the feature graph after feature extraction through the second feature extraction network to obtain global features. Before step 101, the following steps may also be included:

[0061] X1. Reconstruct the training data through the variable automatic encoder module to obtain a reconstructed image;

[0062] X2. Determine a residual image using the training data and the reconstructed image;

[0063] X3. Determine a combined image according to the reconstructed image and the residual image;

[0064] X4. Extract features from the combined image using the deep feature extraction module to obtain training features;

[0065] X5. Adjust the model parameters of the first feature extraction network through the training features and the preset loss function to obtain the second feature extraction network.

[0066] Among them, the preset loss function can be preset or system default, for example, the preset loss function can be a triplet loss function. In a specific implementation, the electronic device can reconstruct the training data through the variable autoencoder module VAE to obtain a reconstructed image, and can further determine the residual image through the training data and the reconstructed image, and determine the combined image according to the reconstructed image and the residual image. The backbone network of the deep feature extraction module can be a ResNet-50 backbone. The combined image is then subjected to feature extraction through the deep feature extraction module to obtain training features, and finally the model parameters of the first feature extraction network are adjusted through the training features and the preset loss function to obtain the second feature extraction network.

[0067] In an embodiment of the present application, the feature extraction network can be composed of two modules: a self-supervised residual generation module and a deep feature extraction. In a specific implementation, the input image can be passed through a VAE-based reconstruction module to remove specific details, and then the reconstructed image is subtracted from the input image to form a residual image containing specific details. Subsequently, a convex combination of the input and the residual (with a trainable parameter α) is calculated and passed through the re-id backbone for deep feature extraction. The entire network is trained with triples and cross entropy losses and separated by a batch normalization layer (BN Neck).

[0068] Further, optionally, the above step X3, determining the combined image according to the reconstructed image and the residual image, may include the following steps:

[0069] X31, determining the first feature point distribution density of the reconstructed image;

[0070] X32, determining the second feature point distribution density of the training data i;

[0071] X33. Determine a ratio between the first feature point distribution density and the second feature point distribution density;

[0072] X34, dividing the reconstructed image into a plurality of regions, determining the distribution density of each feature point in the plurality of regions, and obtaining a plurality of feature point distribution densities;

[0073] X35, determining a target mean square error according to the distribution density of the plurality of feature points;

[0074] X36. Determine a target fine-tuning coefficient corresponding to the target mean square error according to a preset mapping relationship between the mean square error and the fine-tuning coefficient;

[0075] X37, fine-tuning the ratio according to the target fine-tuning coefficient to obtain a convex parameter;

[0076] X38. Determine the combined image based on the convex parameters, the reconstructed image and the residual image.

[0077] In a specific implementation, the electronic device can determine the first feature point distribution density of the reconstructed image, specifically, the total number of feature points of the reconstructed image and the image area, and the ratio between the total number of feature points and the image area is used as the feature point distribution density. Similarly, the electronic device can also determine the second feature point distribution density of the training data i, and then determine the ratio between the first feature point distribution density and the second feature point distribution density, where the ratio = first feature point distribution density / second feature point distribution density.

[0078] Furthermore, the reconstructed image can be divided into multiple regions, the distribution density of each feature point in the multiple regions is determined, and multiple feature point distribution densities are obtained, and the target mean square error is determined based on the multiple feature point distribution densities. The mean square error reflects the correlation between regions, that is, it reflects the difference in changes between image neighborhoods. The electronic device can also pre-store a mapping relationship between a preset mean square error and a fine-tuning coefficient. Then, according to the mapping relationship between the preset mean square error and the fine-tuning coefficient, the target fine-tuning coefficient corresponding to the target mean square error can be determined, and then fine-tuning is performed based on the target fine-tuning coefficient comparison value to obtain the convex parameter α. The specific calculation formula is as follows:

[0079] Convex parameter = (1 + target fine-tuning coefficient) * ratio

[0080] Finally, the electronic device can determine a combined image according to the convex parameter, the reconstructed image and the residual image, that is, the combined image = convex parameter * reconstructed image + residual image.

[0081] 102. Determine target information and target tracking trajectory based on the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image.

[0082] Among them, the above step 102 can be implemented in the detection network or separately. Through the detection network, not only the confidence feature map and position feature map of the i-th frame image can be obtained, but also the position increment feature map and reid feature vector can be obtained, which helps to improve the calculation speed and improve the real-time processing capability.

[0083] Optionally, the above step 102, determining the target information and the target tracking trajectory according to the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image, may include the following steps:

[0084] 21. According to the score value of each point in the confidence feature map, a target score value within a preset range is selected, and target information of the i-th frame image is determined according to the score value, wherein the target information includes a target position and a reid feature corresponding to the target score value in the i-th frame image;

[0085] 22. Estimate the tracking position of the next frame of the i-th frame according to the target position and the position increment feature map;

[0086] 23. Obtain a reference tracking trajectory of the (i-1)th frame, and update the reference tracking trajectory according to the tracking position to obtain the target tracking trajectory.

[0087] In a specific implementation, the above preset range can be set by the user or by the system default. The electronic device can output a target score value within a preset range according to the score value of each point in the confidence feature map, and then determine the target information of the i-th frame image according to the score value. Since the score value corresponds to the position of the pixel point, the corresponding reid feature can also be determined based on the position. The target information may include the target position and reid feature corresponding to the target score value in the i-th frame image. Further, the electronic device can estimate the tracking position of the next frame according to the current target position and the position increment feature map. The target tracking trajectory is initialized by the target information obtained in the previous frame and the estimated tracking position of the next frame. The target tracking trajectory may include the position sequence, reid feature and estimated tracking position of the target in all frames. That is, the electronic device can obtain the reference tracking trajectory of the i-1th frame, and update the reference tracking trajectory according to the tracking position to obtain the target tracking trajectory, so that the trajectory update can be achieved.

[0088] 103. Input the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0089] Among them, in the embodiment of the present application, the electronic device can adopt an end-to-end target tracking network to integrate similarity calculation and data association into one network.

[0090] In a specific implementation, based on the above-mentioned embodiment of the present application, the current frame image and the previous frame target heat map can be input into the detection network, and the current frame confidence feature map, position feature map, position increment feature map, and reid feature vector map can be output. Multi-target detection information and multi-target tracking trajectories are input into the tracking network, and the best matching matrix is ​​output, and then the tracking result is obtained, and the tracking result can be displayed.

[0091] Optionally, the above step 103, inputting the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs, may include the following steps:

[0092] 31. According to the target information and the target tracking trajectory, the tracking network is used to associate row data of all targets in the current frame with targets in all tracking channels to determine a distance matrix;

[0093] 32. Perform a series of operations based on the distance matrix to obtain the optimal allocation matrix, the series of operations including: row expansion, convolution operation, column expansion, convolution operation, FC layer, and Sigmoid layer.

[0094] In a specific implementation, the electronic device can realize row data association of all targets in the current frame with all tracking channel targets through the tracking network according to the target information and the target tracking trajectory, and can realize many-to-many, that is, multiple targets correspond to multiple tracking channel targets, to determine the distance matrix, and then, perform a series of operations according to the distance matrix to obtain the best allocation matrix, the series of operations include: row expansion, convolution operation, column expansion, convolution operation, fully connected (FC) layer, Sigmoid layer, the value in the best allocation matrix represents which tracking target the detected target belongs to

[0095] Specifically, Figure 1D As shown, the electronic device starts from the second frame image, inputs the target information (e.g., target position, re-id features) and target tracking trajectory into the tracking network, calculates the distance matrix of the multi-target detection and multi-target tracking channel trajectory of the current frame, and then flattens it by rows, performs convolution, and then flattens it by columns, performs convolution, and outputs the optimal allocation matrix through the FC layer and the Sigmoid layer.

[0096] Further, optionally, before obtaining the thermal map of the i-th frame image and the i-1-th frame image in the video sequence in step 101, the following steps may also be included:

[0097] A1. Acquire a sample image, wherein the sample image includes a target area and a background area;

[0098] A2. Enhance the target area to obtain multiple target images, and train the detection network based on the multiple target images, wherein the enhancement includes at least one of the following: scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, and optical transformation.

[0099] In the specific implementation, when training the detection network, the target moves in continuous frames, which can be regarded as the same target undergoing multiple different data enhancements, so static images can be used for training. For static target data enhancement, data enhancement includes scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, optical transformation, etc., to simulate the target motion phenomenon. Then select a background image, rotate the target, perform optical perturbations, and Gaussian noise on the target, and then paste it at different positions of the background image to simulate the motion phenomenon of pedestrians, vehicles, and other targets. That is, the electronic device can obtain a sample image, which includes a target area and a background area, and can also enhance the target area to obtain multiple target images, and the enhancement includes at least one of the following: scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, and optical transformation. The static detection training data set can generate a motion tracking training set through data enhancement, and then, samples in various postures can be obtained more accurately.

[0100] Specifically, when training the detection network, you can input a frame of image, perform data enhancement on the target, perform convolution, BN and activation operations, etc., then extract features through the backbone network, and after the convolution layer, BN layer and activation operation, you can output the confidence map feature map, position feature map, position increment feature map, and reid feature vector map. You can also calculate the binary cross entropy loss of the confidence feature map, the distance loss between the target position and the predicted position feature map, the position increment feature map regression loss, and the distance loss of the target reid feature. Then, use these losses to optimize the model parameters of the detection network to improve the detection accuracy of the detection network.

[0101] Furthermore, when training the tracking network, a public tracking data set can be used. First, the data set is tested to obtain the incremental position and reid feature vector of the marked target, and then the best matching matrix is ​​output through the tracking network. The loss of tracking accuracy (whether it is a false alarm, missed target, identity switch) and the loss of tracking precision (whether the marked box and the predicted box match) can also be calculated. The number of false alarms, missed targets, and identity switches is counted as the loss function value of tracking accuracy. The tracking accuracy loss function can select the mean square error loss function, etc., and then the model parameters of the tracking network are optimized through the loss function to improve the tracking accuracy of the tracking network.

[0102] In the embodiment of the present application, the face image set may include multiple face images. The face image set may include a large number of face images. The electronic device may obtain the face image set to be processed from a cloud server or locally.

[0103] In a possible example, when the target is a human face, the above step A1, obtaining a sample image, may include the following steps:

[0104] A1. Obtain an initial face image set;

[0105] A2. performing image quality evaluation on each face image in the face image set to obtain a plurality of face image quality evaluation values;

[0106] A3. Select a facial image quality evaluation value greater than a preset image quality evaluation value from the multiple facial image quality evaluation values, and use the facial image corresponding to the facial image as the sample image.

[0107] Among them, in the embodiment of the present application, the preset image quality evaluation value can be pre-stored in the electronic device, which can be set by the user or by system default.

[0108] In a specific implementation, the electronic device can obtain an initial face image set, and can use at least one image quality evaluation index to evaluate the image quality of each face image in the face image set, and obtain multiple face image quality evaluation values. The image quality evaluation index can be at least one of the following: face deviation, face integrity, clarity, feature point distribution density, average gradient, information entropy, signal-to-noise ratio, etc., which are not limited here. Furthermore, the electronic device can select a face image quality evaluation value greater than a preset image quality evaluation value from multiple face image quality evaluation values, and use the corresponding face image as a sample image. Among them, the face deviation is the deviation between the face angle in the image and the face angle of the front face, and the face integrity is the ratio between the area of ​​the face in the image and the area of ​​the complete face.

[0109] In a possible example, the above step A2, performing image quality evaluation on each face image in the face image set to obtain multiple face image quality evaluation values, may include the following steps:

[0110] A21, obtaining a target face deviation degree of face image j, a target face completeness degree of the face image j, a target feature point distribution density and a target information entropy of the face image j, wherein the face image j is any face image in the face image set;

[0111] A22, when the target face deviation is greater than a preset deviation and the target face integrity is greater than a preset integrity, determining a target first reference evaluation value corresponding to the target face deviation according to a mapping relationship between a preset face deviation and a first reference evaluation value;

[0112] A23, determining a target second reference evaluation value corresponding to the target face integrity according to a preset mapping relationship between the face integrity and the second reference evaluation value;

[0113] A24, according to the preset mapping relationship between the feature point distribution density and the weight pair, determine the target weight pair corresponding to the target feature point distribution density, the target weight pair includes a target first weight and a target second weight, the target first weight is the weight corresponding to the first reference evaluation value, and the target second weight is the weight corresponding to the second reference evaluation value;

[0114] A25, performing a weighted operation according to the target first weight, the target second weight, the target first reference evaluation value, and the target second reference evaluation value to obtain a first reference evaluation value;

[0115] A26. Determine a first image quality evaluation value corresponding to the target feature point distribution density according to a preset mapping relationship between the feature point distribution density and the image quality evaluation value;

[0116] A27, determining a target image quality deviation value corresponding to the target information entropy according to a preset mapping relationship between information entropy and image quality deviation value;

[0117] A28. Obtain a first shooting parameter of the face image j;

[0118] A29. Determine a target optimization coefficient corresponding to the first shooting parameter according to a preset mapping relationship between the shooting parameter and the optimization coefficient;

[0119] A30, adjusting the first image quality evaluation value according to the target optimization coefficient and the target image quality deviation value to obtain a second reference evaluation value;

[0120] A31, obtaining target environment parameters corresponding to the face image j;

[0121] A32. Determine a target weight coefficient pair corresponding to the target environmental parameter according to a preset mapping relationship between the environmental parameter and the weight coefficient pair, wherein the target weight coefficient pair includes a target first weight coefficient and a target second weight coefficient, wherein the target first weight coefficient is a weight coefficient corresponding to the first reference evaluation value, and the target second weight coefficient is a weight coefficient corresponding to the second reference evaluation value;

[0122] A33. Perform a weighted operation based on the target first weight coefficient, the target second weight coefficient, the first reference evaluation value, and the second reference evaluation value to obtain the facial image quality evaluation value of the facial image j.

[0123] Among them, in the embodiment of the present application, the preset deviation degree and the preset integrity degree can be set by the user or by the system default, and only when both are within a certain range can the face recognition be successful. The electronic device can pre-store the mapping relationship between the preset face deviation degree and the first reference evaluation value, the mapping relationship between the preset face integrity degree and the second reference evaluation value, and the mapping relationship between the preset feature point distribution density and the weight pair. The weight pair may include a first weight and a second weight, the sum of the first weight and the second weight is 1, the first weight is the weight corresponding to the first reference evaluation value, and the second weight is the weight corresponding to the second reference evaluation value. The electronic device can also pre-store the mapping relationship between the preset feature point distribution density and the image quality evaluation value, the mapping relationship between the preset information entropy and the image quality deviation value, the mapping relationship between the preset shooting parameters and the optimization coefficient, and the mapping relationship between the preset environmental parameters and the weight coefficient pair. Among them, the weight coefficient pair may include a first weight coefficient and a second weight coefficient, the first weight coefficient is the weight coefficient corresponding to the first reference evaluation value, the second weight coefficient is the weight coefficient corresponding to the second reference evaluation value, and the sum of the first weight coefficient and the second weight coefficient is 1.

[0124] Among them, the value range of the image quality evaluation value can be 0 to 1, or, it can also be 0 to 100. The image quality deviation value can be a positive real number, for example, 0 to 1, or, it can also be greater than 1. The value range of the optimization coefficient can be between -1 and 1, for example, the optimization coefficient can be -0.1 to 0.1. In the embodiment of the present application, the shooting parameters can be at least one of the following: exposure time, shooting mode, sensitivity ISO, white balance parameters, focal length, focus, area of ​​interest, etc., which are not limited here. The environmental parameters can be at least one of the following: ambient brightness, ambient temperature, ambient humidity, weather, atmospheric pressure, magnetic field interference intensity, etc., which are not limited here.

[0125] In a specific implementation, taking face image j as an example, face image j is any face image in a face image set, and the electronic device can obtain the target face deviation degree of face image j, the target face completeness of face image j, the target feature point distribution density of face image j and the target information entropy, wherein the target feature point distribution density can be the ratio between the total number of feature points of face image j and the area of ​​face image j.

[0126] Furthermore, when the target face deviation is greater than the preset deviation and the target face integrity is greater than the preset integrity, the electronic device can determine the target first reference evaluation value corresponding to the target face deviation according to the mapping relationship between the preset face deviation and the first reference evaluation value, and can also determine the target second reference evaluation value corresponding to the target face integrity according to the mapping relationship between the preset face integrity and the second reference evaluation value, and determine the target weight pair corresponding to the target feature point distribution density according to the mapping relationship between the preset feature point distribution density and the weight pair, the target weight pair includes a target first weight and a target second weight, the target first weight is the weight corresponding to the first reference evaluation value, and the target second weight is the weight corresponding to the second reference evaluation value, and then, a weighted operation can be performed based on the target first weight, the target second weight, the target first reference evaluation value and the target second reference evaluation value to obtain a first reference evaluation value, and the specific calculation formula is as follows:

[0127] First reference evaluation value = target first reference evaluation value * target first weight + target second reference evaluation value * target second weight

[0128] Furthermore, the quality of the image can be evaluated holistically from the perspective of the face and the completeness of the face.

[0129] Further, the electronic device may determine the first image quality evaluation value corresponding to the target feature point distribution density according to the mapping relationship between the preset feature point distribution density and the image quality evaluation value, and determine the target image quality deviation value corresponding to the target information entropy according to the mapping relationship between the preset information entropy and the image quality deviation value. The electronic device may determine the target image quality deviation value corresponding to the target information entropy according to the mapping relationship between the preset information entropy and the image quality deviation value. When generating an image, some noise is generated due to external (weather, light, angle, jitter, etc.) or internal (system, GPU) reasons. These noises will have some impact on the image quality. Therefore, the image quality can be adjusted to a certain extent to ensure an objective evaluation of the image quality.

[0130] Furthermore, the electronic device may also obtain a first shooting parameter of the target face image, and determine a target optimization coefficient corresponding to the first shooting parameter according to a mapping relationship between a preset shooting parameter and an optimization coefficient. The shooting parameter setting may also have a certain impact on the image quality evaluation. Therefore, it is necessary to determine the impact component of the shooting parameter on the image quality. Finally, the first image quality evaluation value is adjusted according to the target optimization coefficient and the target image quality deviation value to obtain a second reference evaluation value, wherein the second reference evaluation value can be obtained according to the following formula:

[0131] When the image quality evaluation value is a percentage system, the specific calculation formula is as follows:

[0132] Second reference evaluation value = (first image quality evaluation value + target image quality deviation value) * (1 + target optimization coefficient)

[0133] When the image quality evaluation value is a percentage, the specific calculation formula is as follows:

[0134] Second reference evaluation value = first image quality evaluation value * (1 + target image quality deviation value) * (1 + target optimization coefficient)

[0135] Further, the electronic device can obtain the target environment parameters corresponding to the face image j, and determine the target weight coefficient pair corresponding to the target environment parameters according to the mapping relationship between the preset environment parameters and the weight coefficient pair, the target weight coefficient pair includes a target first weight coefficient and a target second weight coefficient, the target first weight coefficient is the weight coefficient corresponding to the first reference evaluation value, and the target second weight coefficient is the weight coefficient corresponding to the second reference evaluation value. Then, a weighted operation can be performed based on the target first weight coefficient, the target second weight coefficient, the first reference evaluation value, and the second reference evaluation value to obtain the face image quality evaluation value of the face image j. The specific calculation formula is as follows:

[0136] The face image quality evaluation value of face image j = first reference evaluation value * target first weight coefficient + second reference evaluation value * target second weight coefficient

[0137] In this way, the image quality can be objectively evaluated by combining internal and external environmental factors, shooting setting factors, and the influence of face angle and completeness, which helps to improve the accuracy of face image quality evaluation.

[0138] It can be seen that the target tracking method described in the embodiment of the present application, the electronic device includes a target tracking system, the target tracking system includes a detection network and a tracking network, obtains the heat map of the i-th frame image and the i-1-th frame image in the video sequence, and inputs the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, i is an integer greater than or equal to 2, and the target information and the target tracking trajectory are determined according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, and the target information and the target tracking trajectory are input into the tracking network to obtain the optimal allocation matrix, and the numerical value in the optimal allocation matrix represents the tracking target to which the target belongs. The detection network can realize the confidence feature map, position feature map, position increment feature map and reid feature map based on the heat map of the previous frame and the current frame image. The detection network can accurately detect the target and has a re-identification function, and can accurately grasp the target trajectory and target information, thereby realizing accurate classification of the target, which helps to improve the tracking accuracy.

[0139] With the above Figure 1B For the embodiments shown, please refer to Figure 2 , Figure 2 is a flow chart of a target tracking method provided by an embodiment of the present application, which is applied to Figure 1A The electronic device shown includes a target tracking system, the target tracking system includes a detection network and a tracking network, and the target tracking method includes:

[0140] 201. Acquire a sample image, where the sample image includes a target area and a background area.

[0141] 202. Enhance the target area to obtain multiple target images, and train the detection network based on the multiple target images, wherein the enhancement includes at least one of the following: scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, and optical transformation.

[0142] 203. Obtain heat maps of the i-th frame image and the i-1-th frame image in the video sequence, input the heat maps of the i-th frame image and the i-1-th frame image into the detection network, and obtain a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2.

[0143] 204. Determine target information and target tracking trajectory according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image.

[0144] 205. Input the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0145] The specific description of the above steps 201 to 205 can refer to the above Figure 1B The corresponding steps of the described target tracking method will not be repeated here.

[0146] It can be seen that the target tracking method described in the embodiment of the present application, the electronic device includes a target tracking system, the target tracking system includes a detection network and a tracking network, on the one hand, it can generate a motion tracking training set through data enhancement based on a static detection training data set, on the other hand, the detection network can realize a confidence feature map, a position feature map, a position increment feature map and a reid feature map based on the heat map of the previous frame and the current frame image. The detection network can accurately detect the target and has a re-identification function, can accurately grasp the target trajectory and target information, and then realize accurate target classification, which helps to improve tracking accuracy.

[0147] In accordance with the above embodiment, please refer to Figure 3 , Figure 3 : is a structural diagram of an electronic device provided in an embodiment of the present application. As shown in the figure, the device includes a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor. In the embodiment of the present application, the electronic device includes a target tracking system, the target tracking system includes a detection network and a tracking network, and the program includes instructions for executing the following steps:

[0148] Obtain a heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2;

[0149] Determine target information and target tracking trajectory according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image;

[0150] The target information and the target tracking trajectory are input into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0151] It can be seen that the electronic device described in the embodiment of the present application includes a target tracking system, which includes a detection network and a tracking network. The heat map of the i-th frame image and the i-1-th frame image in the video sequence is obtained, and the heat map of the i-th frame image and the i-1-th frame image is input into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2. The target information and the target tracking trajectory are determined according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, and the target information and the target tracking trajectory are input into the tracking network to obtain the optimal allocation matrix. The numerical values ​​in the optimal allocation matrix represent the tracking target to which the target belongs. The detection network can realize the confidence feature map, position feature map, position increment feature map and reid feature map based on the heat map of the previous frame and the current frame image. The detection network can accurately detect the target and has a re-identification function, and can accurately grasp the target trajectory and target information, thereby realizing accurate classification of the target, which helps to improve the tracking accuracy.

[0152] Optionally, in the aspect of inputting the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, the above program includes instructions for performing the following steps:

[0153] The i-th frame image is processed successively through a convolution layer, a BN layer and an activation layer, and the first intermediate result is input into a backbone network for feature extraction, and the extracted features are used to respectively determine the confidence feature map, the position feature map and the reid feature vector of the i-th frame image, wherein the confidence feature map of the i-th frame image is also used to optimize the position feature map, and the position feature map is also used to optimize the reid feature vector, wherein the reid feature vector is obtained by a CosFace loss function, and the map of the reid feature vector is a random value between 0 and 1;

[0154] The heat map of the i-1th frame image is processed successively by a convolution layer, a BN layer and an activation layer to obtain a second intermediate result, and a convolution operation is performed on the second intermediate result and the extracted features, and the position increment feature map is determined based on the result of the convolution operation. The position feature map in turn optimizes the position increment feature map during the back propagation process. The position increment feature map is obtained by the Softmax loss function, and the maps of the initialized position increments are all 0 during training.

[0155] Optionally, after inputting the first intermediate result into the backbone network for feature extraction, the program includes instructions for performing the following steps:

[0156] Perform global feature extraction on the feature map obtained after feature extraction to obtain global features;

[0157] Performing local area detection on the feature map to obtain a region of interest;

[0158] Divide the global feature into S×S grids, where S is an integer greater than 1;

[0159] Projecting the region of interest onto the S×S grids, extracting a local feature vector of each projected grid using local average pooling to obtain local features; the local features in turn optimize the global feature representation during back propagation;

[0160] The global features and the local features are aggregated to obtain extracted features.

[0161] Optionally, in the aspect of determining the target information and the target tracking trajectory according to the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image, the above program includes instructions for executing the following steps:

[0162] According to the score value of each point in the confidence feature map, a target score value in a preset range is selected, and target information of the i-th frame image is determined according to the score value, wherein the target information includes a target position and a reid feature corresponding to the target score value in the i-th frame image;

[0163] Estimate the tracking position of the next frame of the i-th frame according to the target position and the position increment feature map;

[0164] A reference tracking trajectory of the i-1th frame is obtained, and the reference tracking trajectory is updated according to the tracking position to obtain the target tracking trajectory.

[0165] Optionally, in the aspect of inputting the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the numerical values ​​in the optimal allocation matrix represent the tracking target to which the target belongs, the program includes instructions for executing the following steps:

[0166] According to the target information and the target tracking trajectory, the tracking network is used to associate row data of all targets in the current frame with targets in all tracking channels to determine a distance matrix;

[0167] A series of operations are performed according to the distance matrix to obtain the optimal allocation matrix, and the series of operations include: row expansion, convolution operation, column expansion, convolution operation, FC layer, and Sigmoid layer.

[0168] Optionally, before obtaining the thermal map of the i-th frame image and the i-1-th frame image in the video sequence, the program further includes instructions for executing the following steps:

[0169] Acquire a sample image, wherein the sample image includes a target area and a background area;

[0170] The target area is enhanced to obtain multiple target images, and the detection network is trained based on the multiple target images, wherein the enhancement includes at least one of the following: scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, and optical transformation.

[0171] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should be easily aware that, in combination with the units and algorithm steps of each example described in the embodiment provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0172] The embodiment of the present application can divide the functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0173] Figure 4 4 is a functional unit composition block diagram of a target tracking device 400 involved in an embodiment of the present application. The device 400 is applied to an electronic device, the electronic device includes a target tracking system, the target tracking system includes a detection network and a tracking network, the device 400 includes: a detection unit 401, a determination unit 402 and a tracking unit 403, wherein,

[0174] The detection unit 401 is used to obtain the heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2;

[0175] The determining unit 402 is used to determine the target information and the target tracking trajectory according to the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image;

[0176] The tracking unit 403 is used to input the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, and the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs.

[0177] It can be seen that the target tracking device described in the embodiment of the present application is applied to an electronic device, and the electronic device includes a target tracking system, and the target tracking system includes a detection network and a tracking network. The heat map of the i-th frame image and the i-1-th frame image in the video sequence is obtained, and the heat map of the i-th frame image and the i-1-th frame image is input into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2, and the target information and the target tracking trajectory are determined according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, and the target information and the target tracking trajectory are input into the tracking network to obtain the optimal allocation matrix, and the numerical value in the optimal allocation matrix represents the tracking target to which the target belongs. The detection network can realize the confidence feature map, position feature map, position increment feature map and reid feature map based on the heat map of the previous frame and the current frame image. The detection network can accurately detect the target and has a re-identification function, and can accurately grasp the target trajectory and target information, thereby realizing accurate classification of the target, which helps to improve the tracking accuracy.

[0178] Optionally, in the aspect of inputting the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image, the detection unit 401 is specifically used for:

[0179] The i-th frame image is processed successively through a convolution layer, a BN layer and an activation layer, and the first intermediate result is input into a backbone network for feature extraction, and the extracted features are used to respectively determine the confidence feature map, the position feature map and the reid feature vector of the i-th frame image, wherein the confidence feature map of the i-th frame image is also used to optimize the position feature map, and the position feature map is also used to optimize the reid feature vector, wherein the reid feature vector is obtained by a CosFace loss function, and the map of the reid feature vector is a random value between 0 and 1;

[0180] The heat map of the i-1th frame image is processed successively by a convolution layer, a BN layer and an activation layer to obtain a second intermediate result, and a convolution operation is performed on the second intermediate result and the extracted features, and the position increment feature map is determined based on the result of the convolution operation. The position feature map in turn optimizes the position increment feature map during the back propagation process. The position increment feature map is obtained by the Softmax loss function, and the maps of the initialized position increments are all 0 during training.

[0181] Optionally, after inputting the first intermediate result into the backbone network for feature extraction, the detection unit 401 is further specifically used for:

[0182] Perform global feature extraction on the feature map obtained after feature extraction to obtain global features;

[0183] Performing local area detection on the feature map to obtain a region of interest;

[0184] Divide the global feature into S×S grids, where S is an integer greater than 1;

[0185] Projecting the region of interest onto the S×S grids, extracting a local feature vector of each projected grid using local average pooling to obtain local features; the local features in turn optimize the global feature representation during back propagation;

[0186] The global features and the local features are aggregated to obtain extracted features.

[0187] Optionally, in determining the target information and the target tracking trajectory according to the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image, the determining unit 402 is specifically used to:

[0188] According to the score value of each point in the confidence feature map, a target score value in a preset range is selected, and target information of the i-th frame image is determined according to the score value, wherein the target information includes a target position and a reid feature corresponding to the target score value in the i-th frame image;

[0189] Estimate the tracking position of the next frame of the i-th frame according to the target position and the position increment feature map;

[0190] A reference tracking trajectory of the i-1th frame is obtained, and the reference tracking trajectory is updated according to the tracking position to obtain the target tracking trajectory.

[0191] Optionally, in the aspect of inputting the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the numerical values ​​in the optimal allocation matrix represent the tracking targets to which the targets belong, the tracking unit 403 is specifically configured to:

[0192] According to the target information and the target tracking trajectory, the tracking network is used to associate row data of all targets in the current frame with targets in all tracking channels to determine a distance matrix;

[0193] A series of operations are performed according to the distance matrix to obtain the optimal allocation matrix, and the series of operations include: row expansion, convolution operation, column expansion, convolution operation, FC layer, and Sigmoid layer.

[0194] Optionally, before obtaining the heat map of the i-th frame image and the i-1-th frame image in the video sequence, the device 400 is further specifically used for:

[0195] Acquire a sample image, wherein the sample image includes a target area and a background area;

[0196] The target area is enhanced to obtain multiple target images, and the detection network is trained based on the multiple target images, wherein the enhancement includes at least one of the following: scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, and optical transformation.

[0197] It can be understood that the functions of each program module of the target tracking device of this embodiment can be specifically implemented according to the method in the above method embodiment, and its specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0198] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0199] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0200] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0201] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0203] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0206] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0207] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A target tracking method, characterized in that: Applied to an electronic device, the electronic device includes a target tracking system, the target tracking system includes a detection network and a tracking network, and the method includes: Obtain a heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain a confidence feature map, a position feature map, a position increment feature map and a reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2; Determine target information and target tracking trajectory according to the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image; Inputting the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs; The step of inputting the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image comprises: The i-th frame image is processed successively through a convolution layer, a BN layer and an activation layer, and the first intermediate result is input into a backbone network for feature extraction, and the extracted features are used to respectively determine the confidence feature map, the position feature map and the reid feature vector of the i-th frame image, wherein the confidence feature map of the i-th frame image is also used to optimize the position feature map, and the position feature map is also used to optimize the reid feature vector, wherein the reid feature vector is obtained by a CosFace loss function, and the map of the reid feature vector is a random value between 0 and 1; The heat map of the i-1th frame image is processed successively by a convolution layer, a BN layer and an activation layer to obtain a second intermediate result, and a convolution operation is performed on the second intermediate result and the extracted features, and the position increment feature map is determined based on the result of the convolution operation. The position feature map in turn optimizes the position increment feature map during the back propagation process. The position increment feature map is obtained by the Softmax loss function, and the maps of the initialized position increments are all 0 during training.

2. The method according to claim 1, characterized in that After inputting the first intermediate result into the backbone network for feature extraction, the method further includes: Perform global feature extraction on the feature map obtained after feature extraction to obtain global features; Performing local area detection on the feature map to obtain a region of interest; Divide the global feature into S×S grids, where S is an integer greater than 1; Projecting the region of interest onto the S×S grids, extracting a local feature vector of each projected grid using local average pooling to obtain local features; the local features in turn optimize the global feature representation during back propagation; The global features and the local features are aggregated to obtain extracted features.

3. The method according to claim 1 or 2, characterized in that: The determining of target information and target tracking trajectory based on the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image includes: According to the score value of each point in the confidence feature map, a target score value in a preset range is selected, and target information of the i-th frame image is determined according to the score value, wherein the target information includes a target position and a reid feature corresponding to the target score value in the i-th frame image; Estimate the tracking position of the next frame of the i-th frame according to the target position and the position increment feature map; A reference tracking trajectory of the i-1th frame is obtained, and the reference tracking trajectory is updated according to the tracking position to obtain the target tracking trajectory.

4. The method according to claim 1 or 2, characterized in that: The target information and the target tracking trajectory are input into the tracking network to obtain an optimal allocation matrix, wherein the numerical values ​​in the optimal allocation matrix represent the tracking target to which the target belongs, including: According to the target information and the target tracking trajectory, the tracking network is used to associate row data of all targets in the current frame with targets in all tracking channels to determine a distance matrix; A series of operations are performed according to the distance matrix to obtain the optimal allocation matrix, and the series of operations include: row expansion, convolution operation, column expansion, convolution operation, FC layer, and Sigmoid layer.

5. The method according to claim 1 or 2, characterized in that: Before obtaining the thermal map of the i-th frame image and the i-1-th frame image in the video sequence, the method further includes: Acquire a sample image, wherein the sample image includes a target area and a background area; The target area is enhanced to obtain multiple target images, and the detection network is trained based on the multiple target images, wherein the enhancement includes at least one of the following: scaling, translation, rotation, perspective transformation, color transformation, distortion transformation, cropping, and optical transformation.

6. A target tracking device, characterized in that: Applied to electronic equipment, the electronic equipment includes a target tracking system, the target tracking system includes a detection network and a tracking network, the device includes: a detection unit, a determination unit and a tracking unit, wherein: The detection unit is used to obtain the heat map of the i-th frame image and the i-1-th frame image in the video sequence, input the heat map of the i-th frame image and the i-1-th frame image into the detection network, and obtain the confidence feature map, position feature map, position increment feature map and reid feature vector of the i-th frame image, where i is an integer greater than or equal to 2; The determining unit is used to determine the target information and the target tracking trajectory according to the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image; The tracking unit is used to input the target information and the target tracking trajectory into the tracking network to obtain an optimal allocation matrix, wherein the values ​​in the optimal allocation matrix represent the tracking target to which the target belongs; Wherein, in the aspect of inputting the heat map of the i-th frame image and the i-1-th frame image into the detection network to obtain the confidence feature map, the position feature map, the position increment feature map and the reid feature vector of the i-th frame image, the detection unit is specifically used for: The i-th frame image is processed successively through a convolution layer, a BN layer and an activation layer, and the first intermediate result is input into a backbone network for feature extraction, and the extracted features are used to respectively determine the confidence feature map, the position feature map and the reid feature vector of the i-th frame image, wherein the confidence feature map of the i-th frame image is also used to optimize the position feature map, and the position feature map is also used to optimize the reid feature vector, wherein the reid feature vector is obtained by a CosFace loss function, and the map of the reid feature vector is a random value between 0 and 1; The heat map of the i-1th frame image is processed successively by a convolution layer, a BN layer and an activation layer to obtain a second intermediate result, and a convolution operation is performed on the second intermediate result and the extracted features, and the position increment feature map is determined based on the result of the convolution operation. The position feature map in turn optimizes the position increment feature map during the back propagation process. The position increment feature map is obtained by the Softmax loss function, and the maps of the initialized position increments are all 0 during training.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the program comprises instructions for executing the steps in the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 5.

Images