Target object tracking method, device and equipment and computer readable storage medium

Abstract

Embodiments of the present application provide a target object tracking method and device, equipment and computer readable storage medium, relating to the technical field of artificial intelligence. The method comprises: receiving a target tracking request; in response to the target tracking request, performing feature extraction on an initial template, a current cumulative template and a to-be-tracked region corresponding to a to-be-tracked image respectively, and correspondingly obtaining initial feature embedding, current cumulative feature embedding and to-be-tracked feature embedding; performing cross-correlation processing on the template aggregated feature embedding and the to-be-tracked feature embedding obtained by feature extraction, to obtain a similarity between each sub-region and a target object; determining a tracking result of the target tracking request in at least two sub-regions according to the similarity, and outputting the tracking result. Through the embodiments of the present application, long-range changes in the appearance of the target object can be captured at a relatively low time complexity, and the tracking performance and accuracy of the target object are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of Internet, and relate to but are not limited to a target object tracking method, device, equipment and computer readable storage medium. BACKGROUND

[0002] Target tracking technology is a hot spot in the field of computer vision, and is applied to human-computer interaction, intelligent video monitoring and traffic detection.

[0003] At present, the target tracking technology usually uses optical flow, recurrent neural network (RNN) or long short-term memory network (LSTM) to update the template when tracking the target, and usually propagates the template from the previous frame to the current frame to realize template matching.

[0004] Although the current target tracking method can use time information to improve tracking performance, the current target tracking method can only capture the changes in the appearance of the target in a short time, and ignores the long-term changes in the appearance of the target. Since the target is prone to serious occlusion or large deformation in a short time, which will reduce the target tracking performance, the accuracy of the current target tracking method is low. SUMMARY

[0005] Embodiments of the present application provide a target object tracking method, device, equipment and computer readable storage medium, which relate to the technical field of artificial intelligence. Since the initial template and the current cumulative template are considered at the same time, real-time tracking of the target object is realized, so that the long-range changes in the appearance of the target object can be captured at a relatively low time complexity, and the tracking performance of the target object and the accuracy of tracking the target object are improved.

[0006] The technical scheme of the embodiments of the present application is as follows:

[0007] The embodiments of the present application provide a target object tracking method, comprising:

[0008] receiving a target tracking request; the target tracking request includes an initial template, a current cumulative template and a to-be-tracked image, and the initial template includes the target object;

[0009] In response to the target tracking request, the initial template, the current cumulative template and the to-be-tracked region corresponding to the to-be-tracked image are respectively subjected to feature extraction, and initial feature embedding, current cumulative feature embedding and to-be-tracked feature embedding are obtained correspondingly; the to-be-tracked region includes at least two sub-regions;

[0010] aggregate the initial feature embedding and the current accumulated feature embedding to obtain a template aggregated feature embedding;

[0011] correlate the template aggregated feature embedding and the to-be-tracked feature embedding to obtain a similarity between each of the sub-regions and the target object;

[0012] determine a tracking result of the target tracking request in the at least two sub-regions according to the similarity, and output the tracking result.

[0013] Embodiments of the present application provide a target object tracking device, comprising:

[0014] a receiving module configured to receive a target tracking request; the target tracking request comprising an initial template, a current accumulated template and a to-be-tracked image, the initial template comprising the target object;

[0015] a feature extraction module configured to, in response to the target tracking request, perform feature extraction on a to-be-tracked region corresponding to the initial template, the current accumulated template and the to-be-tracked image respectively, and obtain an initial feature embedding, a current accumulated feature embedding and a to-be-tracked feature embedding correspondingly; the to-be-tracked region comprising at least two sub-regions;

[0016] an aggregation processing module configured to aggregate the initial feature embedding and the current accumulated feature embedding to obtain a template aggregated feature embedding;

[0017] a cross-correlation processing module configured to correlate the template aggregated feature embedding and the to-be-tracked feature embedding to obtain a similarity between each of the sub-regions and the target object;

[0018] a determining module configured to determine a tracking result of the target tracking request in the at least two sub-regions according to the similarity, and output the tracking result.

[0019] Embodiments of the present application provide a target object tracking device, comprising:

[0020] a memory configured to store executable instructions; and a processor configured to execute the executable instructions stored in the memory to implement the target object tracking method described above.

[0021] Embodiments of the present application provide a computer readable storage medium storing executable instructions, configured to cause a processor to execute the executable instructions to implement the target object tracking method described above.

[0022] The embodiments of the present application have the following beneficial effects: considering the initial template and the current cumulative template at the same time, performing aggregation processing on the initial feature embedding of the initial template and the current cumulative feature embedding of the current cumulative template to obtain a template aggregated feature embedding, and performing cross-correlation processing on the template aggregated feature embedding and the to-be-tracked feature embedding of the to-be-tracked region to determine the similarity between each sub-region in the to-be-tracked region and the target object, thereby realizing real-time tracking of the target object. In this way, since the initial template and the current cumulative template are considered at the same time, real-time tracking of the target object is realized, the long-range change of the appearance of the target object can be captured at a relatively low time complexity, and the tracking performance of the target object and the accuracy of tracking the target object are improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is an optional architecture schematic diagram of a target object tracking system provided by the embodiments of the present application;

[0024] Figure 2 is a structure schematic diagram of a server provided by the embodiments of the present application;

[0025] Figure 3 is an optional flow schematic diagram of a target object tracking method provided by the embodiments of the present application;

[0026] Figure 4 is an optional flow schematic diagram of a target object tracking method provided by the embodiments of the present application;

[0027] Figure 5 is an optional flow schematic diagram of a target object tracking method provided by the embodiments of the present application;

[0028] Figure 6 is an optional flow schematic diagram of a target object tracking method provided by the embodiments of the present application;

[0029] Figure 7 is an optional flow schematic diagram of a target object tracking method provided by the embodiments of the present application;

[0030] Figure 8 is an optional flow schematic diagram of a template diffusion network training method provided by the embodiments of the present application;

[0031] Figure 9 is a flowchart of a fast and robust target object tracking method based on a template diffusion network provided by the embodiments of the present application. DETAILED DESCRIPTION

[0032] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0033] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as understood by those skilled in the art to which the embodiments of the present application belong. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] In order to better understand the target object tracking method provided in the embodiments of the present application, first, the target object tracking method in the related art is described:

[0035] For a visual tracking task, in the currently mainstream visual tracking method based on a twin network, a large number of pictures are first sampled in a data set to perform offline training on the twin network, and then a template of an initial frame and a search region of a current frame are used to perform matching through the twin network to locate the position of the target when online tracking. However, visual tracking is a dynamic process, and the visual tracking method based on the twin network cannot adapt to the appearance change of the target object by only using the template of the initial frame (i.e., the initial template) for matching.

[0036] In the related art, in order to adapt to the appearance change of the target in the tracking process, a graph convolutional tracking method (GCT, Graph Convolutional Tracking) fully utilizes the spatiotemporal structure information of the template to construct a graph convolutional network to realize tracking of the target object; a flow tracking method (FlowTrack) introduces an optical flow and a spatiotemporal attention mechanism in a twin network to effectively aggregate the template to realize tracking of the target object. In order to improve the online adaptability of the twin network, a dynamic twin network (such as DSiamM) learns a fast transformation from the initial template to the last frame template, and proposes a dynamic twin network to adapt to the appearance change of the target object in the tracking process; a memory tracking network (such as MemTrack) learns a dynamic memory network, and captures the appearance change of the target by reading and writing the memory module in the tracking process. In order to effectively update the template, a gradient guided network (GradNet) proposes a gradient guided twin network to update the template by using gradient information; in the update network (such as UpdateNet and LeUpdate), researchers respectively use a convolutional neural network (CNN, Convolutional Neural Networks) and an RNN to update the template of the twin network.

[0037] The current tracking methods usually use optical flow, RNN or LSTM for template updating, and usually propagate the template from the previous frame to the current frame to realize template matching. Although these tracking methods can utilize temporal information to improve tracking performance, they still have some defects. For example, the optical flow can only reflect the motion information in the continuous frame, but the reflected motion information will accumulate errors over time. The RNN usually uses the truncated backpropagation technique to reduce the computational complexity and alleviate the gradient vanishing problem, which limits the ability of the RNN to model long-term dependencies. The LSTM often uses a gating mechanism to alleviate the gradient vanishing problem, but the gating mechanism still has the problem of gradient explosion. In addition, these tracking methods can only capture the changes in the appearance of the target in a short period of time, and ignore the long-term temporal changes in the appearance of the target. Since the target is prone to severe occlusion or large deformation in a short period of time, this can cause the tracking performance to decline.

[0038] According to the above analysis, in order to solve at least one problem existing in the target object tracking method in the related art, an embodiment of the present application proposes a fast and robust target object tracking method based on a template diffusion network (TDNet), which can capture the appearance changes of the target over time with relatively low computational complexity. The TDNet proposed in the embodiment of the present application embeds a feature aggregation module (FAM) into a deep twin network to adapt to the changes in the appearance of the target object over time during the tracking process. The FAM is essentially a simple and effective attention module that can effectively establish the correspondence between the template feature embeddings in the channel and space, and establish the long-range dependency of the initial template and the current cumulative template. In this way, the TDNet proposed in the embodiment of the present application can effectively handle long-range target object appearance changes with relatively low computational cost. Experiments conducted on standard datasets show that, compared with several latest real-time trackers, the proposed TDNet can significantly improve the tracking performance while maintaining a speed of 120 frames per second (FPS).

[0039] The target object tracking method provided by the embodiment of the present application first receives a target tracking request. The target tracking request includes an initial template, a current cumulative template and a to-be-tracked image, and the initial template includes the target object. Then, in response to the target tracking request, the initial template, the current cumulative template and the to-be-tracked image are respectively subjected to feature extraction, and initial feature embeddings, current cumulative feature embeddings and to-be-tracked feature embeddings are obtained. Then, the initial feature embeddings and the current cumulative feature embeddings are subjected to aggregation processing to obtain template aggregation feature embeddings. The template aggregation feature embeddings and the to-be-tracked feature embeddings are subjected to cross-correlation processing to obtain the similarity between each sub-region in the to-be-tracked region and the target object. Finally, according to the similarity, a tracking result of the target tracking request is determined in at least two sub-regions, and the tracking result is output. In this way, since the initial template and the current cumulative template are considered at the same time, real-time tracking of the target object is realized, the long-range changes in the appearance of the target object can be captured with relatively low time complexity, and the tracking performance of the target object and the accuracy of the target object tracking are improved.

[0040] The following describes an exemplary application of the target object tracking device of the embodiments of the present application. In one implementation, the target object tracking device provided by the embodiments of the present application can be implemented as any terminal such as a notebook computer, a tablet computer, a desktop computer, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), a smart robot, a smart video monitoring or traffic monitoring device, and the like. In another implementation, the target object tracking device provided by the embodiments of the present application can also be implemented as a server. The following describes an exemplary application when the target object tracking device is implemented as a server.

[0041] Referring to Figure 1 , Figure 1 is an optional architecture schematic diagram of the target object tracking system 10 provided by the embodiments of the present application. To realize tracking of a target object in a video file, the target object tracking system 10 provided by the embodiments of the present application includes a terminal 100, a network 200, and a server 300. The terminal 100 runs a video playing application or has a video recording unit, plays a real-time recorded video or a pre-recorded video through the video playing application, and tracks a target object in the video through the method of the embodiments of the present application.

[0042] In the method of the embodiments of the present application, after the video is acquired, the terminal 100 determines a current frame of the video as a to-be-tracked region, and acquires a frame image including the target object selected by a user in the video, and determines the frame image as an initial template. The terminal 100 sends the to-be-tracked region and the initial template to the server 300 through the network 200. The server 300 determines a current cumulative template according to a tracking result of a previous time, and respectively extracts features from the acquired initial template, the current cumulative template, and the to-be-tracked region, to correspondingly obtain an initial feature embedding, a current cumulative feature embedding, and a to-be-tracked feature embedding. Then, the initial feature embedding and the current cumulative feature embedding are aggregated to obtain a template aggregated feature embedding. The template aggregated feature embedding and the to-be-tracked feature embedding are cross-correlated to obtain a similarity between each sub-region in the to-be-tracked region and the target object. The target sub-region where the target object is located is determined in at least two sub-regions according to the similarity. The target sub-region is determined as a tracking result of the target object. The tracking result is marked, and a marked image marked with the tracking result is sent to the terminal 100. The terminal 100 displays the marked image marked with the tracking result on a current interface. Through the method of the embodiments of the present application, the target object tracking can be performed on each frame image in the video, so that the target object can be marked and displayed on the terminal 100 in real time.

[0043] The target object tracking method provided by the embodiments of the present application further relates to the field of artificial intelligence and can be implemented by using computer vision technology and machine learning technology in the field of artificial intelligence. The computer vision technology (CV) is a science of how to make a machine "see". In further words, the computer vision technology refers to using a camera and a computer to replace human eyes to identify, track and measure a target and further perform image processing, so that the computer processing becomes an image more suitable for human eye observation or transmission to an instrument detection. As a scientific discipline, the computer vision technology researches related theories and technologies and attempts to establish an artificial intelligence system capable of obtaining information from images or multidimensional data. The computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and the like. The computer vision technology also includes common face recognition, fingerprint recognition and other biometric recognition technologies. In the embodiments of the present application, the computer vision technology is used to track a target object in any frame of image in a video, so as to determine a position of the target object in the image or determine whether the target object exists in the image.

[0044] Machine learning (ML) is a multi-field cross discipline, which involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The machine learning is specially used to study how a computer simulates or realizes a learning behavior of a human being to obtain new knowledge or skills or reorganize an existing knowledge structure to constantly improve the performance of the computer. The machine learning is the core of the artificial intelligence and is a fundamental approach to enabling the computer to have intelligence. The machine learning is applied to various fields of the artificial intelligence. The machine learning and deep learning usually include artificial neural network, belief network, reinforcement learning, transfer learning, inductive learning, and inductive learning. In the embodiments of the present application, the machine learning technology is used to train a template diffusion network and optimize network parameters.

[0045] Figure 2 FIG. 3 is a structural schematic diagram of a server 300 provided by the embodiments of the present application, Figure 2 The server 300 shown in FIG. 3 includes at least one processor 310, a memory 350, at least one network interface 320 and a user interface 330. The various components in the server 300 are coupled together by a bus system 340. It can be understood that the bus system 340 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 340 in Figure 2 .

[0046] The processor 310 can be an integrated circuit chip having a processing capability of signals, such as a general purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general purpose processor can be a microprocessor or any conventional processor.

[0047] The user interface 330 includes one or more output devices 331 enabling presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 330 also includes one or more input devices 332, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0048] The memory 350 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 350 optionally includes one or more storage devices remotely located from the processor 310. The memory 350 comprises a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 350 described in embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below.

[0049] The operating system 351 includes a system program for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0050] The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including Bluetooth, wireless fidelity (WiFi), and universal serial bus (USB), etc.

[0051] The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0052] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 2 A target object tracking apparatus 354 stored in the memory 350 is shown, which can be a target object tracking apparatus in the server 300, and can be software in the form of programs and plug-ins, etc., including the following software modules: a receiving module 3541, a feature extraction module 3542, an aggregation processing module 3543, a cross-correlation processing module 3544, and a determination module 3545, which are logical and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0053] In some other embodiments, the apparatus provided by the embodiments of the present application can be implemented in hardware, for example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the target object tracking method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0054] The target object tracking method provided by the embodiments of the present application will be described below in conjunction with exemplary applications and implementations of the server 300 provided by the embodiments of the present application. Referring to Figure 3 , Figure 3 is an optional flowchart of the target object tracking method provided by the embodiments of the present application, which will be described in conjunction with the steps shown. Figure 3

[0055] In step S301, a target tracking request is received.

[0056] In the embodiments of the present application, a video playing application or a target tracking application is running on the terminal. During the playing of the video on the terminal, the user can send a target tracking request to the server through the terminal, and the target tracking request is used to request tracking of a target object in any frame of the video, so as to present the position change of the target object in the video.

[0057] The target tracking request includes an initial template, a current cumulative template, and a to-be-tracked image, and the initial template includes the target object to be tracked.

[0058] ​Here, the initial template is a selection template input by a user, and the initial template can be a certain frame image in a video corresponding to the image to be tracked, or can be any frame image in a preset image library. The initial template includes a target object, and the user can determine the target object to be tracked in advance, and then select the initial template including the target object.

[0059] The target object can be any type of movable or deformable object, for example, the target object can be a person, an animal, a vehicle, or the like, or the target object can also be a hand. The method of the embodiments of the present application can be applied to any type of target object tracking system, for example, the system can be a road traffic monitoring system, and the corresponding target object is a vehicle; or the system can be a video monitoring system, and the corresponding target object can be a person; or the system can be a gesture monitoring and recognition system, and the corresponding target object can be a hand, and the motion and deformation of the hand in the video are tracked and monitored to realize intelligent gesture recognition.

[0060] The current cumulative template is a template determined according to the historical cumulative template in a historical time period before the current time, that is, a template obtained after gradually accumulating the historical template. The current cumulative template considers the long-range changes in the appearance of the target object over time.

[0061] The to-be-tracked region is an image region corresponding to the image to be tracked, and the image to be tracked can be any frame of video in the video to be tracked. The to-be-tracked region includes at least two sub-regions, and the image size between each two sub-regions can be the same or different, or the image size of each sub-region is the same as the size of the initial target and the size of the current cumulative template.

[0062] In step S302, in response to the target tracking request, feature extraction is performed on the initial template, the current cumulative template, and the to-be-tracked region corresponding to the image to be tracked, and initial feature embedding, current cumulative feature embedding, and to-be-tracked feature embedding are obtained correspondingly.

[0063] Here, the to-be-tracked region includes at least two sub-regions. In the embodiments of the present application, any feature extraction network can be used to perform feature extraction on the input initial template, current cumulative template, and to-be-tracked region, to obtain initial feature embedding, current cumulative feature embedding, and to-be-tracked feature embedding correspondingly.

[0064] In step S303, the initial feature embedding and the current cumulative feature embedding are aggregated to obtain a template aggregation feature embedding.

[0065] Here, the aggregation processing refers to the accumulated features obtained by aggregating the features related to the channel and the features related to the space in the initial feature embedding and the current cumulative feature embedding, respectively.

[0066] In step S304, cross-correlation processing is performed on the template aggregated feature embedding and the to-be-tracked feature embedding to obtain a similarity between each sub-region in the to-be-tracked region and the target object.

[0067] Here, the cross-correlation processing refers to calculating the similarity between each sub-region in the to-be-tracked region and the target object or calculating a probability value of the presence of the target object in each sub-region in the to-be-tracked region by using the template aggregated feature embedding and the to-be-tracked feature embedding.

[0068] In step S305, a tracking result of the target tracking request is determined in at least two sub-regions according to the similarity, and the tracking result is output.

[0069] Here, the sub-region with the highest similarity is determined as the target sub-region, or the sub-region with a similarity greater than a similarity threshold is determined as the target sub-region, or all sub-regions in the to-be-tracked region are sorted in descending order of similarity to form a sub-region sequence, and the first sub-region in the sub-region sequence is determined as the target sub-region.

[0070] In the embodiment of the present application, the target sub-region is determined as the tracking result of the target object, that is, the target sub-region is determined as the tracking result of the target tracking request. After the target sub-region is determined, the target sub-region can also be marked in the to-be-tracked region, for example, the target sub-region can be marked by using a bounding box, or the regions other than the target sub-region are blurred, and only the target sub-region is highlighted.

[0071] The target object tracking method provided in the embodiment of the present application can track a certain target object in a video, and in the process of playing the video, the target sub-region where the determined target object is located is marked in each video frame. Thus, in the process of continuously playing the video, the user sees only the target object in the bounding box, and the position of the bounding box changes continuously according to the change of the position of the target object.

[0072] The target object tracking method provided in the embodiment of the present application simultaneously considers the initial template and the current cumulative template, performs aggregation processing on the initial feature embedding of the initial template and the current cumulative feature embedding of the current cumulative template to obtain the template aggregated feature embedding, and performs cross-correlation processing on the template aggregated feature embedding and the to-be-tracked feature embedding of the to-be-tracked region to determine the similarity between each sub-region in the to-be-tracked region and the target object, thereby realizing real-time tracking of the target object. Thus, since the initial template and the current cumulative template are simultaneously considered, real-time tracking of the target object is realized, the long-range change of the appearance of the target object can be captured at a relatively low time complexity, and the tracking performance of the target object and the accuracy of tracking the target object are improved.

[0073] In some embodiments, the target object tracking method can be applied to a target object tracking system, which at least comprises a terminal and a server, and the terminal is provided with a video acquisition device, for example, a traffic monitoring terminal. In order to track a target object in a video acquired by the terminal in real time, the method of the embodiment of the application can be implemented by the following steps, as shown in Figure 4 The method comprises the following steps:

[0074] In step S401, the terminal acquires a to-be-tracked video.

[0075] The to-be-tracked video comprises a target object. When the method of the embodiment of the application is applied to a traffic detection system, the target object can be a vehicle with a certain license plate.

[0076] In step S402, the terminal acquires a selection operation of a user.

[0077] In some embodiments, the selection operation corresponds to any video frame in the video, and the selection operation is used to select the video frame as an initial template for real-time tracking of the target object. For example, a video frame in which the target object first appears in the currently acquired video can be selected as the initial template, or any video frame in which the target object is clearly photographed can be selected as the initial template, so as to realize real-time tracking of the target object.

[0078] In other embodiments, the selection operation can not correspond to any video frame in the video. Through the selection operation, a preset image in an image library is selected, and it is determined whether the target object in the preset image exists in the currently acquired video through real-time tracking of the currently acquired video. For example, photos of different vehicles (including license plate numbers) can be stored in the image library. When tracking a target vehicle, the image of the target vehicle can be selected as the initial template to track the target vehicle in real time.

[0079] In step S403, the terminal sends a target tracking request to the server, and the target tracking request comprises the initial template and the video, or the target tracking request comprises the initial template and the currently acquired video frame.

[0080] Here, if it is necessary to track the target object in a pre-recorded video, the initial template and the video can be sent to the server, and the server tracks the target object appearing in each frame of the video; if it is necessary to track the target object in the currently acquired video frame, the video frame acquired in real time can be sent to the server in sequence to track the target object.

[0081] In step S404, the server determines a region corresponding to the current video frame or the current captured video frame in the video as a to-be-tracked region. The to-be-tracked region includes at least two sub-regions.

[0082] In step S405, the server determines a current accumulated template according to the tracking result of the previous video frame.

[0083] Here, the current accumulated template is an accumulated template accumulated with the template information in the historical video frame and the tracking result of the previous video frame, that is, the current accumulated template not only considers the template information in the historical video frame, but also considers the position of the target object determined by the tracking result of the previous frame.

[0084] In step S406, the server respectively extracts features from the initial template, the current accumulated template, and the to-be-tracked region, and correspondingly obtains an initial feature embedding, a current accumulated feature embedding, and a to-be-tracked feature embedding.

[0085] In step S407, the server aggregates the initial feature embedding and the current accumulated feature embedding to obtain a template aggregated feature embedding.

[0086] In step S408, the server performs cross-correlation processing on the template aggregated feature embedding and the to-be-tracked feature embedding to obtain a similarity between each sub-region in the to-be-tracked region and the target object.

[0087] In step S409, the server determines a target sub-region in which the target object is located from the at least two sub-regions according to the similarity.

[0088] In step S410, the server determines the target sub-region as the tracking result of the target object.

[0089] In step S411, the server marks the target sub-region in the current video frame or the current captured video frame in the video according to the tracking result to obtain a marked image.

[0090] Here, the target sub-region can be circled in the current video frame or the current captured video frame, or the image of the target sub-region can be highlighted, for example, the brightness of the target sub-region is increased, to obtain the marked image.

[0091] In step S412, the server sends the marked image to the terminal.

[0092] In step S413, the terminal displays a tracking video formed by the marked image on the current interface.

[0093] The target object tracking provided in the embodiments of the present application is that the terminal sends the collected video to the server, the server tracks the target object in the video in real time, the initial template and the current cumulative template are considered simultaneously in the tracking process, the long-range change of the appearance of the target object can be captured under a relatively low time complexity, the tracking performance of the target object is improved, the accuracy of tracking the target object is improved, the tracking result is displayed on the terminal in real time to the user, the use experience of the user is improved, and the convenience of monitoring illegal shooting of the traffic monitoring system is provided.

[0094] Based on Figure 3 , Figure 5 is an optional flowchart of a target object tracking method provided in the embodiments of the present application, as shown in Figure 5 In some embodiments, step S303 can be implemented by the following steps:

[0095] Step S501, performing spatial feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain spatial aggregation features.

[0096] Here, the spatial feature aggregation processing refers to aggregating spatially similar semantic features in the initial feature embedding and the current cumulative feature embedding corresponding to the initial template and the current cumulative template, in order to highlight the feature embedding of the to-be-tracked image remaining in the template, and suppress the feature embedding corresponding to the background area of the to-be-tracked image.

[0097] In some embodiments, the determination of the spatial aggregation features in step S501 can be implemented by the following steps:

[0098] Step S5011, obtaining a first spatial embedding function of the initial feature embedding and a second spatial embedding function of the current cumulative feature embedding. Step S5012, determining a spatial similarity matrix between the initial feature embedding and the current cumulative feature embedding according to the first spatial embedding function and the second spatial embedding function. Step S5013, obtaining a third spatial embedding function of the current cumulative feature embedding and a first adjustment parameter. Step S5014, determining the value of the spatial aggregation features on each feature channel according to the spatial similarity matrix, the third spatial embedding function and the first adjustment parameter, to obtain the spatial aggregation features.

[0099] Here, the third spatial embedding function is different from the second spatial embedding function.

[0100] In some embodiments, the determination of the spatial aggregation features in steps S5011 to S5014 can be implemented by the following formulas (1-1) and (1-2):

[0101] (1-1);

[0102] (1-2);

[0103] wherein, denotes a spatial similarity matrix , and ; denotes an initial feature embedding, wherein, ; denotes a current accumulated feature embedding, wherein, ; denotes a real number field, denotes a template height, denotes a template width, denotes a number of feature channels; denotes an initial feature embedding ; denotes a first spatial embedding function of an n-th feature channel of a current accumulated feature embedding ; denotes a size of a spatial similarity matrix ;

[0104] denotes a value of an m-th feature channel of an output spatial aggregated feature ; is a first adjustment parameter for adjusting a value of a spatial aggregated feature ; denotes a second spatial embedding function of an n-th feature channel of a current accumulated feature embedding , and is different from ; denotes a feature value of an m-th feature channel of an initial feature embedding .

[0105] In step S502, a channel feature aggregation processing is performed on the initial feature embedding and the current accumulated feature embedding to obtain a channel aggregated feature.

[0106] Here, since the feature channels of the target object and its surrounding background in the template (including the initial template and the current accumulated template) are interrelated, i.e., some feature channels cooperatively represent similar semantics (e.g., target object, occlusion or background interference), the spatial feature aggregation processing refers to aggregating the feature channels with similar semantics in the initial template and the current accumulated template features.

[0107] In some embodiments, the determination of the channel aggregated feature in step S502 can be implemented by the following steps:

[0108] In step S5021, a first channel embedding function of the initial feature embedding and a second channel embedding function of the current accumulated feature embedding are obtained. In step S5022, a channel similarity matrix between the initial feature embedding and the current accumulated feature embedding is determined according to the first channel embedding function and the second channel embedding function. In step S5023, a third channel embedding function of the current accumulated feature embedding and a second adjustment parameter are obtained. In step S5024, a value of the channel aggregated feature on each feature channel is determined according to the channel similarity matrix, the third channel embedding function and the second adjustment parameter, so as to obtain the channel aggregated feature.

[0109] Here, the third channel embedding function is different from the second channel embedding function.

[0110] In some embodiments, the channel aggregated feature is determined in step S502 The above can be implemented by the following formulas (1-3) and (1-4):

[0111] (1-3) ;

[0112] (1-4) ;

[0113] wherein, denotes an element in the i-th row and the j-th column of the channel similarity matrix ; denotes the initial feature embedding, wherein, ; denotes the current accumulated feature embedding, wherein, ; denotes a real number field, denotes a template height, denotes a template width, denotes a number of feature channels; denotes a channel embedding function of the i-th feature channel of the initial feature embedding ; denotes a first channel embedding function of the j-th feature channel of the current accumulated feature embedding ;

[0114] denotes a value of the i-th feature channel of the output channel aggregated feature ; is a second adjustment parameter for adjusting the value of the channel aggregated feature ; denotes a second channel embedding function of the j-th feature channel of the current accumulated feature embedding , and is different from ; denotes the initial feature embedding the feature value of the i-th feature channel.

[0115] In step S503, the template aggregated feature embedding is determined according to the spatial aggregated feature and the channel aggregated feature.

[0116] In some embodiments, step S503 can be implemented by the following steps:

[0117] In step S5031, an initial feature embedding is obtained.

[0118] In step S5032, elements of each corresponding position of the initial feature embedding, the spatial aggregated feature and the channel aggregated feature are summed to obtain element sums of the corresponding positions.

[0119] In step S5033, the template aggregated feature embedding is determined according to the element sums of all positions.

[0120] Here, the initial feature embedding, the spatial aggregated feature and the channel aggregated feature are three embedding vectors with the same dimension, and when the template aggregated feature embedding is determined, the three embedding vectors are summed according to element positions, that is, elements of the same positions in the three embedding vectors are summed to obtain element sums, and then the obtained element sums are assigned to corresponding positions of the template aggregated feature embedding, so that a template aggregated feature embedding with the same dimension as the initial feature embedding, the spatial aggregated feature and the channel aggregated feature can be obtained.

[0121] Based on Figure 3 , Figure 6 is an optional flowchart of the target object tracking method provided by the embodiments of the present application, and in some embodiments, before the feature extraction on the initial template and the current accumulated template, the method further includes a method for determining the initial template and the current accumulated template, as shown in Figure 6 Before step S301, the method further includes the following steps:

[0122] In step S601, a historical accumulated template of a historical moment before and adjacent to a current moment and a historical template determined according to a tracking result of the historical moment are obtained.

[0123] In step S602, the current accumulated template is determined according to the historical accumulated template and the historical template.

[0124] In some embodiments, step S602 can be implemented by the following formula (1-5):

[0125] (1-5);

[0126] wherein, denotes the current accumulated template; denotes a learning rate; representing the history accumulated template; representing the history template. It should be noted that the current time is t, t-1 represents a history time point before the current time and adjacent to the current time, since the current accumulated template is calculated based on the information of t-1, the current accumulated template is represented as ; t-2 represents another history time point before t-1 and adjacent to t-1.

[0127] In step S603, before online tracking of the to-be-tracked region, the history accumulated template of the history time point before the current time and adjacent to the current time is initialized to obtain an initialized template.

[0128] Here, the initialization stage is to initialize the history accumulated template of the history time point to an initial template for the target tracking process.

[0129] In step S604, the initialized template is determined as the initial template.

[0130] In some embodiments, the initial template and the current accumulated template are aggregated to form an aggregated template, and the template aggregation feature embedding is a feature embedding representation of the aggregated template; based on Figure 3 , Figure 7 is an optional flowchart of the target object tracking method provided by the embodiments of the present application, as shown in Figure 7 Step S304 can be implemented by the following steps:

[0131] In step S701, the sub-feature embedding corresponding to each sub-region in the to-be-tracked feature embedding is determined.

[0132] In step S702, the similarity between each sub-region and the target object is determined by sliding the aggregated template in the to-be-tracked region.

[0133] Here, the similarity between the sub-region corresponding to the to-be-tracked region of the to-be-tracked image and the target object is calculated by sliding the aggregated template to determine the similarity between the sub-region and the target object. When sliding the aggregated template, the sliding step can be the same each time, and the sliding step is smaller than the size of the aggregated template, that is, there is an overlap between the sub-region slid by the previous time and the sub-region slid by the next time. In the embodiments of the present application, the sliding can be started from the boundary position of the to-be-tracked region according to a preset direction.

[0134] In some embodiments, the similarity between each sub-region and the target object can be calculated by the following formula (1-6):

[0135] (1-6);

[0136] wherein, denotes a similarity function between a sub-region x and a target object z, and the similarity score between each sub-region and the target object is calculated by sliding the aggregation template on the region to be tracked; denotes a feature embedding function of the fully convolutional twin network, which is used for feature extraction and feature embedding of the sub-region x or the target object z, wherein, denotes a feature embedding function of the fully convolutional twin network for the target object z, denotes a feature embedding function of the fully convolutional twin network for the sub-region x; denotes a cross-correlation operation, wherein the cross-correlation refers to an infinite integral of multiplying the complex conjugate and reverse translation of two functions, or an infinite integral of multiplying the complex conjugate and translation of the first function in turn with the second function.

[0137] Please continue to refer to Figure 7 After determining the similarity between each sub-region and the target object, the method further comprises the following steps:

[0138] Step S703, determining the response value of each sub-region according to the similarity between each sub-region and the target object; wherein, the higher the similarity, the greater the response value of the corresponding sub-region.

[0139] Step S704, drawing a response map corresponding to the region to be tracked according to the response value.

[0140] The response map not only includes each sub-region in the region to be tracked, but also includes the response value of each sub-region, that is, the response value of each sub-region is added to the corresponding position in the response map, so that the similarity between each sub-region and the target object can be more intuitively seen, and thus the target sub-region can be quickly determined.

[0141] Correspondingly, step S305 can be implemented by the following steps:

[0142] Step S705, determining the sub-region corresponding to the maximum response value in the response map as the target sub-region. Step S706, determining the target sub-region as the tracking result of the target tracking request.

[0143] The target object tracking method provided by the embodiments of the present application can also be implemented by using a template diffusion network trained based on artificial intelligence technology, that is, using the template diffusion network for feature extraction, aggregation processing and cross-correlation processing to obtain the similarity between each sub-region in the region to be tracked and the target object.

[0144] Figure 8is an optional flowchart of the template diffusion network training method provided by the embodiment of the present application, as shown in Figure 8 The training method comprises the following steps:

[0145] In step S801, input the sample initial template, the sample accumulated template and the sample tracking area into the template diffusion network; the sample initial template comprises a sample target object.

[0146] In step S802, perform feature extraction on the sample initial template, the sample accumulated template and the sample tracking area respectively through the feature extraction network in the template diffusion network, and obtain the sample initial feature embedding, the sample accumulated feature embedding and the sample tracking feature embedding correspondingly.

[0147] Here, the template diffusion network can have one feature extraction network, which performs feature extraction on the sample initial template, the sample accumulated template and the sample tracking area respectively, or the template diffusion network can have three feature extraction networks, each of which corresponds to one kind of input data, and performs feature extraction on the sample initial template, the sample accumulated template and the sample tracking area respectively and simultaneously through the three feature extraction networks.

[0148] In step S803, perform spatial feature aggregation processing on the sample initial feature embedding and the sample accumulated feature embedding through the spatial feature aggregation network in the template diffusion network, and obtain the sample spatial aggregation feature.

[0149] The spatial feature aggregation network is used to aggregate the spatially similar semantic features in the sample initial template and the sample accumulated template.

[0150] In step S804, perform channel feature aggregation processing on the sample initial feature embedding and the sample accumulated feature embedding through the channel feature aggregation network in the template diffusion network, and obtain the sample channel aggregation feature.

[0151] The channel feature aggregation network is used to aggregate the feature channels with similar semantics in the sample initial template and the sample accumulated template. The channel feature aggregation network has the same input as the spatial feature aggregation network, and the dimensions of the sample spatial aggregation feature and the sample channel aggregation feature output by the channel feature aggregation network are the same.

[0152] In step S805, perform fusion on the sample spatial aggregation feature and the sample channel aggregation feature through the aggregated feature fusion unit in the template diffusion network, and obtain the sample template aggregation feature embedding.

[0153] Here, when the aggregated feature fusion unit performs feature fusion, the sample initial feature embedding can also be fused into the sample spatial aggregation feature and the sample channel aggregation feature through a skip connection branch, to obtain the sample template aggregation feature embedding.

[0154] In the embodiments of the present application, the sample space aggregated features, the sample channel aggregated features and the sample initial features can be embedded in the sample template aggregated feature embedding, the elements at the same positions are summed bit by bit, and then the sum result is taken as the sample template aggregated feature embedding at the corresponding position elements. By summing all positions, the sample template aggregated feature embedding is generated.

[0155] In step S806, the sample template aggregated feature embedding and the sample tracking feature embedding are cross-correlated by the cross-correlation processing unit in the template diffusion network to obtain the sample similarity between each sub-sample region in the sample tracking region and the sample target object.

[0156] In the embodiments of the present application, the sample template aggregated feature embedding corresponds to the sample aggregated template between the sample initial template and the sample cumulative template, that is, the feature embedding of the sample aggregated template is the sample template aggregated feature embedding. The cross-correlation processing refers to sliding the sample aggregated template on the sample tracking region to calculate the sample similarity score between the corresponding position (i.e., the sub-sample region) and the sample target object.

[0157] In step S807, the sample similarity of each sub-sample region is input into the preset loss model to obtain a loss result.

[0158] Here, the preset loss model is used to compare the sample similarity with the preset similarity to obtain the loss result, wherein the preset similarity can be the similarity between each sub-sample region and the sample target object set by the user in advance.

[0159] In the embodiments of the present application, the preset loss model includes a loss function, by which the difference or distance between the sample similarity and the preset similarity can be calculated, and the above loss result can be determined according to the difference or distance. When the difference or distance between the sample similarity and the preset similarity is larger, it indicates that the difference between the training result of the model and the true value is larger, and further training is needed; when the difference or distance between the sample similarity and the preset similarity is smaller, it indicates that the training result of the model is closer to the true value.

[0160] In step S808, according to the loss result, the parameters in the feature extraction network, the spatial feature aggregation network and the channel feature aggregation network are corrected to obtain the corrected template diffusion network.

[0161] Here, when the difference or distance is greater than the preset threshold, the loss result indicates that the feature extraction network in the current template diffusion network cannot accurately extract features of the sample initial template, the sample accumulated template and the sample tracking region respectively to obtain accurate sample initial feature embedding, sample accumulated feature embedding and sample tracking feature embedding, and / or the spatial feature aggregation network cannot accurately perform spatial feature aggregation processing on the sample initial feature embedding and the sample accumulated feature embedding to obtain accurate sample spatial aggregation features, and / or the channel feature aggregation network cannot accurately perform channel feature aggregation processing on the sample initial feature embedding and the sample accumulated feature embedding to obtain accurate sample channel aggregation features. Therefore, the current template diffusion network needs to be corrected. Then, the parameters in at least one of the feature extraction network, the spatial feature aggregation network and the channel feature aggregation network can be corrected according to the difference or distance until the distance between the sample similarity output by the template diffusion network and the preset similarity satisfies the preset condition, and the corresponding template diffusion network is determined as the trained template diffusion network.

[0162] The training method of the template diffusion network provided by the embodiments of the present application can input the sample initial template, the sample accumulated template and the sample tracking region into the template diffusion network, sequentially perform feature extraction, spatial feature aggregation processing and channel feature aggregation processing through the feature extraction network, the spatial feature aggregation network and the channel feature aggregation network, then perform cross-correlation processing on the sample template aggregation feature embedding and the sample tracking feature embedding, obtain the sample similarity between each sub-sample region in the sample tracking region and the sample target object, input the sample similarity into the preset loss model, and obtain the loss result. Therefore, the parameters in at least one of the feature extraction network, the spatial feature aggregation network and the channel feature aggregation network can be corrected according to the loss result, and the obtained template diffusion network can accurately determine the similarity between each sub-region in the to-be-tracked region and the target object, so as to accurately track the target object in the to-be-tracked region and improve the user experience.

[0163] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0164] The embodiments of the present application provide a target object tracking method, and the embodiments of the present application propose a simple and effective method to capture long-term temporal dependencies, and the method proposed by the embodiments of the present application learns the similarity matrix between arbitrary frame template feature embeddings.

[0165] To ensure short-term tracking consistency and avoid long-term tracking drift, inspired by non-local neural networks, the feature aggregation module proposed in the embodiments of the present application propagates the initial template of the first frame into the current accumulated template maintained in the current frame. The embodiments of the present application insert this feature aggregation module into a deep twin network, and propose an effective template diffusion network (TDNet) for visual tracking. The proposed TDNet can capture long-range changes in target appearance with lower time complexity. The experimental results on the current popular benchmark dataset show that the proposed TDNet tracking method can achieve good performance among similar methods and can run at a real-time speed of 120 FPS.

[0166] Figure 9 is the flowchart of the fast and robust target object tracking method based on the template diffusion network provided by the embodiments of the present application, that is, the flowchart of the target object tracking method, as shown in Figure 9

[0167] First, the feature extraction network 901 (that is, a feature extractor, which can be a fully convolutional AlexNet, for example) is used to encode the initial template , the accumulated template (that is, the current accumulated template) and the search region (that is, the region to be tracked) into corresponding feature embeddings (that is, the initial feature embedding), (that is, the current accumulated feature embedding) and (that is, the feature embedding to be tracked). It should be noted that the accumulated template is updated by the following formula (2-1):

[0168] (2-1) ;

[0169] wherein is the accumulated template of the t-2 frame, is the accumulated template of the t-1 frame, is the template of the t-1 frame based on the tracking result, is the learning rate; wherein C represents the number of channels of the template or the search region. H / H' and W / W' represent the height and width of the template or the search region, respectively.

[0170] Then, input and to two parallel branches of the template diffusion network: the spatial attention branch 902 and the channel attention branch 903, to focus on the feature embedding of the specific target in space and channel. At the same time, input ​The input to the skip connection branch 904 is to preserve the feature embedding of the initial template. The outputs of the spatial attention branch 902, the channel attention branch 903 and the skip connection branch 904 are summed up element-wise 905 to generate a new feature embedding Since the information of the initial template and the accumulated template is effectively fused, the long-range changes of the appearance of the target object can be adapted.

[0171] Finally, the cross-correlation processing 906 is performed on and to measure the similarity so as to generate a response map for target localization, where P and Q represent the height and width of the response map respectively, and the candidate region corresponding to the maximum response value in the response map is the tracking result of the target object.

[0172] The TDNet proposed in the embodiments of the present application is trained end-to-end by using cross-entropy loss. Here, the embodiments of the present application can select any two templates for template aggregation. For offline training, the embodiments of the present application can randomly select two templates (as the initial template and the accumulated template) from a video sequence for training. For online tracking, the embodiments of the present application select the initial template and the accumulated template for template aggregation, because the initial template cropped from the first frame can ensure that the target object is in the template, and the continuously updated accumulated template takes into account the changes in the appearance of the target object over time.

[0173] In some embodiments, the TDNet proposed in the embodiments of the present application can effectively embed the feature aggregation module into a fully convolutional siamese network, where the fully convolutional siamese network learns a similarity function by sliding a template over a search region to calculate the similarity scores of the corresponding positions, and when the entire search region is slid, the response map is composed of the similarity scores of different positions. The similarity calculation is performed on all the candidate regions obtained by sliding (in the search region ) by cross-correlation calculation, where the cross-correlation calculation is shown in the following formula (2-2):

[0174] (2-2);

[0175] wherein represents the cross-correlation operation, represents the feature embedding function of the fully convolutional siamese network, represents the sliding template in the feature embedding function of the fully convolutional siamese network. representing a search region The feature embedding function of the fully convolutional twin network. By using the above formula (2-2), the most similar candidate region in the search region is selected as the tracking result of the target object.

[0176] Although most of the tracking methods based on twin networks can achieve good performance and fast running speed, these methods only use the initial template cropped from the first frame for matching in the tracking process, so they cannot capture the long-range changes of the target appearance over time. However, in the embodiments of the present application, the template feature embedding from different spatial positions, channels and time frames can provide rich information to simulate the appearance changes of the target, such as various rotations, multiple deformations and different degrees of occlusion. For online tracking, the template feature embedding should be adaptively aggregated across time frames in space and channel.

[0177] The TDNet proposed in the embodiments of the present application consists of a fully convolutional twin network and a feature aggregation module, wherein the target of the feature aggregation module of the TDNet is to aggregate the features of the specific target object on different templates, while suppressing the features in the background region that are irrelevant to the target object. Since the non-local operation can capture the long-range dependencies of the target object in an image, the embodiments of the present application can propose two complementary feature aggregation modules based on the non-local operation: a spatial feature aggregation module (SFAM) and a channel feature aggregation module (CFAM), which correspond to the spatial attention branch 902 and the channel attention branch 903 in the TDNet, respectively. Figure 9

[0178] In the channel feature aggregation module, for visual tracking, it is well known that different channels of the features extracted by the fully convolutional twin network usually have a class or target selection effect. The embodiments of the present application believe that the feature channels of the target object and its surrounding background in the template are correlated with each other, that is, some feature channels cooperatively represent similar semantics (such as the target object, the occlusion or the background interference).

[0179] The embodiments of the present application design a CFAM to aggregate the feature channels with similar semantics in the initial template and the cumulative template features. Given the feature embedding of the initial template and the feature embedding of the cumulative template , wherein C is the number of feature channels, H and W are the height and width of the feature map respectively, the embodiments of the present application calculate and ​Channel similarity matrix between :

[0180] (2-3);

[0181] (2-4)

[0182] in, Channel similarity matrix The element in the i-th row and j-th column, and They are and Channel embedding function, Represents the initial feature embedding The channel embedding function of the i-th feature channel. Indicates the current cumulative feature embedding The first channel embedding function of the j-th feature channel. Used to represent The i-th feature channel pair The influence of the j-th feature channel.

[0183] It is the output feature embedding The value of the i-th feature channel (i.e., the output channel aggregation feature); yes Another channel embedding function, Indicates the current cumulative feature embedding The second channel embedding function of the j-th feature channel, and and different; Represents the initial feature embedding The eigenvalue of the i-th feature channel; This is the second adjustment parameter, used to adjust the channel aggregation features. The value, that is, the ability to adjust CFAM. The impact.

[0184] In the spatial feature aggregation module, for target tracking based on a fully convolutional Siamese network, the feature embeddings corresponding to the target region are significantly more important than those in the background region within the template. To highlight the feature embeddings retained in the target region of the template and suppress the feature embeddings corresponding to the background region, this embodiment designs a SFAM to aggregate spatially similar semantic features from the initial template and accumulated template features. The input to SFAM is the same as that of CFAM, which is feature embeddings. (From the initial template) Extraction and feature embedding (From the cumulative template) extracted). Given and , the spatial similarity matrix between and is calculated as follows in Equations (2-5) and (2-6)

[0185] (2-5)

[0186] (2-6)

[0187] where denotes the element in the mth row and nth column of the spatial similarity matrix , and and are the spatial embedding functions of and , respectively, denotes the initial feature embedding , and denotes the first spatial embedding function of the nth feature channel of the current accumulated feature embedding . is used to represent the influence of the mth feature channel of on the nth feature channel of .

[0188] is the value of the mth feature channel of the output feature embedding (i.e., the output spatial aggregated feature); is another spatial embedding function of , and denotes the second spatial embedding function of the nth feature channel of the current accumulated feature embedding . is a first adjustment parameter used to adjust the value of the spatial aggregated feature , i.e., the influence of the SFAM on .

[0189] In the training and testing process of the template diffusion network, the TDNet proposed in the embodiments of the present application needs to be trained offline and tracked online using the TDNet.

[0190] ​In the offline training process, the video frames in GOT10K target tracking dataset are used as training data by embodiments of the present application. GOT10K contains about 10,000 videos, with more than 1.5 million annotated bounding boxes in total, which can be used to train and evaluate twin network-based tracking methods. In each video segment of the target object, three images are randomly selected as training samples, of which the first two frames are regarded as templates, and the remaining one frame is selected as the region to be tracked. Embodiments of the present application use a Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9 and a weight decay of 5.0x10 4 to train the TDNet proposed by embodiments of the present application, with 50 iterations, a minimum batch size of 8, and a learning rate exponentially decaying from 10 2 to 10 5 .

[0191] In the online tracking process, in the initialization stage, the cumulative template is initialized as the initial template, and the initial template and the cumulative template are input into the TDNet. In this case, the feature aggregation module will become a self-attention module. In the tracking stage, not only the initial template is preserved to effectively save the most reliable information, but also the cumulative template is updated at a certain learning rate to consider the appearance change information of the target. After the feature embedding of the generated template is cross-correlated with the feature embedding of the current search region, the tracking result is determined by searching for the maximum response value in the response map. It should be noted that embodiments of the present application are applicable to single-target visual tracking.

[0192] Table 1 is a comparison table of precision, success rate and speed of embodiments of the present application and several other real-time target tracking methods on the OTB100 dataset. Among them, TDNet is the method of embodiments of the present application. As can be seen from Table 1, the TDNet method provided by embodiments of the present application has higher precision and success rate than the current target tracking methods, and the speed is only second to the SiamFC-GK method.

[0193] Table 1

[0194]

[0195] The following continues to illustrate an exemplary structure of the target object tracking device 354 provided by embodiments of the present application implemented as a software module. In some embodiments, as shown in Figure 2 , the software module stored in the target object tracking device 354 of the memory 350 can be a target object tracking device in the server 300, which includes:

[0196] The receiving module 3541 is configured to receive a target tracking request; the target tracking request comprises an initial template, a current cumulative template, and a to-be-tracked image, and the initial template comprises the target object;

[0197] The feature extraction module 3542 is configured to perform feature extraction on a to-be-tracked region corresponding to the initial template, the current cumulative template, and the to-be-tracked image respectively in response to the target tracking request, to obtain an initial feature embedding, a current cumulative feature embedding, and a to-be-tracked feature embedding; the to-be-tracked region comprises at least two sub-regions;

[0198] The aggregation processing module 3543 is configured to perform aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain a template aggregation feature embedding;

[0199] The cross-correlation processing module 3544 is configured to perform cross-correlation processing on the template aggregation feature embedding and the to-be-tracked feature embedding to obtain a similarity between each of the sub-regions and the target object;

[0200] The determining module 3545 is configured to determine a tracking result of the target tracking request in the at least two sub-regions according to the similarity, and output the tracking result.

[0201] In some embodiments, the aggregation processing module is further configured to perform spatial feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain a spatial aggregation feature; perform channel feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain a channel aggregation feature; and determine the template aggregation feature embedding according to the spatial aggregation feature and the channel aggregation feature.

[0202] In some embodiments, the aggregation processing module is further configured to obtain the initial feature embedding; sum elements at each corresponding position of the initial feature embedding, the spatial aggregation feature, and the channel aggregation feature to obtain element sums at the corresponding positions; and determine the template aggregation feature embedding according to the element sums at all positions.

[0203] In some embodiments, the aggregation processing module is further configured to obtain a first spatial embedding function of the initial feature embedding and a second spatial embedding function of the current cumulative feature embedding; determine a spatial similarity matrix between the initial feature embedding and the current cumulative feature embedding according to the first spatial embedding function and the second spatial embedding function; obtain a third spatial embedding function of the current cumulative feature embedding and a first adjustment parameter; and determine values of the spatial aggregation feature on each feature channel according to the spatial similarity matrix, the third spatial embedding function, and the first adjustment parameter, to obtain the spatial aggregation feature.

[0204] In some embodiments, the aggregation processing module is further configured to: obtain a first channel embedding function of the initial feature embedding and a second channel embedding function of the current accumulated feature embedding; determine a channel similarity matrix between the initial feature embedding and the current accumulated feature embedding according to the first channel embedding function and the second channel embedding function; obtain a third channel embedding function of the current accumulated feature embedding and a second adjustment parameter; and determine the value of the channel aggregated feature on each feature channel according to the channel similarity matrix, the third channel embedding function and the second adjustment parameter, to obtain the channel aggregated feature.

[0205] In some embodiments, the apparatus further comprises: an obtaining module configured to obtain a historical accumulated template of a historical time point before and adjacent to the current time point, and a historical template determined according to a tracking result of the historical time point; and a third determining module configured to determine the current accumulated template according to the historical accumulated template and the historical template.

[0206] In some embodiments, the apparatus further comprises: an initialization processing module configured to, in an initialization stage before online tracking of the region to be tracked, perform initialization processing on a historical accumulated template of a historical time point before and adjacent to the current time point, to obtain an initialization template; and an initial template determining module configured to determine the initialization template as the initial template.

[0207] In some embodiments, the initial template and the current accumulated template are aggregated to form an aggregated template, and the template aggregated feature embedding is a feature embedding representation of the aggregated template; and the cross-correlation processing module is further configured to: determine a sub-feature embedding corresponding to each of the sub-regions in the feature embedding to be tracked; and determine the similarity between each of the sub-regions and the target object by sliding the aggregated template in the region to be tracked.

[0208] In some embodiments, the determining module is further configured to: determine a target sub-region in which the target object is located in the at least two sub-regions according to the similarity; and determine the target sub-region as the tracking result of the target tracking request.

[0209] In some embodiments, the apparatus further comprises: a response value determining module configured to determine a response value of each of the sub-regions according to the similarity between each of the sub-regions and the target object; wherein the higher the similarity, the greater the response value of the corresponding sub-region; and a drawing module configured to draw a response map corresponding to the region to be tracked according to the response values; correspondingly, the determining module is further configured to: determine, in the response map, a sub-region corresponding to a maximum response value as the target sub-region.

[0210] In some embodiments, the apparatus further comprises a processing module configured to perform the feature extraction, the aggregation processing and the cross-correlation processing using a template propagation network to obtain a similarity between each of the sub-regions in the tracking region and the target object; wherein the template propagation network is trained by: inputting a sample initial template, a sample accumulated template and a sample tracking region into the template propagation network; the sample initial template comprising a sample target object; performing feature extraction on the sample initial template, the sample accumulated template and the sample tracking region by a feature extraction network in the template propagation network, to obtain a sample initial feature embedding, a sample accumulated feature embedding and a sample tracking feature embedding, respectively; performing spatial feature aggregation processing on the sample initial feature embedding and the sample accumulated feature embedding by a spatial feature aggregation network in the template propagation network, to obtain a sample spatial aggregated feature; performing channel feature aggregation processing on the sample initial feature embedding and the sample accumulated feature embedding by a channel feature aggregation network in the template propagation network, to obtain a sample channel aggregated feature; fusing the sample spatial aggregated feature and the sample channel aggregated feature by an aggregated feature fusion unit in the template propagation network, to obtain a sample template aggregated feature embedding; performing cross-correlation processing on the sample template aggregated feature embedding and the sample tracking feature embedding by a cross-correlation processing unit in the template propagation network, to obtain a sample similarity between each of the sub-sample regions in the sample tracking region and the sample target object; inputting the sample similarity of each of the sub-sample regions into a preset loss model, to obtain a loss result; and modifying parameters in the feature extraction network, the spatial feature aggregation network and the channel feature aggregation network according to the loss result, to obtain a modified template propagation network.

[0211] It should be noted that the description of the apparatus embodiments of the present application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments, and thus is not described in detail. For technical details not disclosed in the apparatus embodiments, please refer to the description of the method embodiments for understanding.

[0212] The computer program product or computer program provided in the embodiments of the present application comprises computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above in the embodiments of the present application.

[0213] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor executes the method provided by the embodiment of the present application, for example, as shown in the method. Figure 3 The method shown.

[0214] In some embodiments, the storage medium can be a computer readable storage medium, for example, a ferroelectric memory (FRAM, Ferromagnetic Random Access Memory), a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read Only Memory), an electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read Only Memory), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM, Compact Disk-Read Only Memory) and the like memory; and can also be various devices including one or any combination of the above memories.

[0215] In some embodiments, the executable instructions can be in the form of a program, software, software module, script or code, written in any form of programming language (including a compiled or interpreted language, or a declarative or procedural language), and can be deployed in any form, including being deployed as an independent program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0216] As an example, the executable instructions can but not necessarily correspond to a file in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a hyper text markup language (HTML, Hyper Text Markup Language) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts). As an example, the executable instructions can be deployed to execute on one computing device, or on multiple computing devices located in one place, or on multiple computing devices distributed in multiple places and interconnected through a communication network.

[0217] The above merely provides an example of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A target object tracking method characterized by, The method comprises: receiving a target tracking request; the target tracking request comprises an initial template, a current cumulative template, and a to-be-tracked image, and the initial template comprises a target object; in response to the target tracking request, performing feature extraction on a to-be-tracked region corresponding to the initial template, the current cumulative template, and the to-be-tracked image respectively, and obtaining initial feature embedding, current cumulative feature embedding, and to-be-tracked feature embedding correspondingly; the to-be-tracked region comprises at least two sub-regions; performing spatial feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain spatial aggregated features; and performing channel feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain channel aggregated features; summing elements at each corresponding position of the initial feature embedding, the spatial aggregated features, and the channel aggregated features to obtain element sums at corresponding positions; and determining a template aggregated feature embedding according to the element sums at all positions; determining a sub-feature embedding corresponding to each of the sub-regions in the to-be-tracked feature embedding; and determining a similarity between each of the sub-regions and the target object by sliding an aggregated template in the to-be-tracked region; wherein the initial template and the current cumulative template are aggregated to form the aggregated template; determining a tracking result of the target tracking request in the at least two sub-regions according to the similarity, and outputting the tracking result.

2. The method of claim 1, wherein, The spatial feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain spatial aggregated features comprises: obtaining a first spatial embedding function of the initial feature embedding and a second spatial embedding function of the current cumulative feature embedding; determining a spatial similarity matrix between the initial feature embedding and the current cumulative feature embedding according to the first spatial embedding function and the second spatial embedding function; obtaining a third spatial embedding function of the current cumulative feature embedding and a first adjustment parameter; determining values of the spatial aggregated features on each feature channel according to the spatial similarity matrix, the third spatial embedding function, and the first adjustment parameter, to obtain the spatial aggregated features.

3. The method of claim 1, wherein, The channel feature aggregation processing on the initial feature embedding and the current cumulative feature embedding to obtain channel aggregated features comprises: obtaining a first channel embedding function of the initial feature embedding and a second channel embedding function of the current cumulative feature embedding; determining a channel similarity matrix between the initial feature embedding and the current cumulative feature embedding according to the first channel embedding function and the second channel embedding function; obtaining a third channel embedding function of the current cumulative feature embedding and a second adjustment parameter; determining values of the channel aggregated features on each feature channel according to the channel similarity matrix, the third channel embedding function, and the second adjustment parameter, to obtain the channel aggregated features.

4. The method of claim 1, wherein, The method further comprises: obtaining a historical cumulative template of a historical moment before and adjacent to a current moment, and a historical template determined according to a tracking result of the historical moment; According to the historical accumulated template and the historical template, the current accumulated template is determined.

5. The method of claim 1, wherein, The method further comprises: In an initialization stage before online tracking of the region to be tracked, a historical accumulated template of a historical moment before and adjacent to a current moment is initialized to obtain an initialized template; The initialized template is determined as the initial template.

6. The method of claim 1, wherein, The template aggregation feature embedding is a feature embedding representation of the aggregated template.

7. The method of claim 1, wherein, According to the similarity, a target sub-region where the target object is located is determined in the at least two sub-regions; and The target sub-region is determined as the tracking result of the target tracking request. The method further comprises:

8. The method of claim 7, wherein, According to the similarity between each sub-region and the target object, a response value of each sub-region is determined; wherein the higher the similarity, the greater the response value of the corresponding sub-region; According to the response value, a response map corresponding to the region to be tracked is drawn; According to the similarity, a target sub-region where the target object is located is determined in the at least two sub-regions, comprising: In the response map, a sub-region corresponding to the maximum response value is determined as the target sub-region. The method further comprises: using a template diffusion network to perform the feature extraction, the aggregation processing and the cross-correlation processing to obtain the similarity between each sub-region in the region to be tracked and the target object; 9. The method according to any one of claims 1 to 8, characterized in that, The template diffusion network is trained by the following steps: Sample initial templates, sample accumulated templates and sample tracking regions are input into the template diffusion network; the sample initial templates include sample target objects; Feature extraction is performed on the sample initial templates, the sample accumulated templates and the sample tracking regions by a feature extraction network in the template diffusion network, to correspondingly obtain sample initial feature embeddings, sample accumulated feature embeddings and sample tracking feature embeddings; Spatial feature aggregation processing is performed on the sample initial feature embeddings and the sample accumulated feature embeddings by a spatial feature aggregation network in the template diffusion network, to obtain sample spatial aggregation features; Channel feature aggregation processing is performed on the sample initial feature embeddings and the sample accumulated feature embeddings by a channel feature aggregation network in the template diffusion network, to obtain sample channel aggregation features; The sample spatial aggregation features and the sample channel aggregation features are fused by an aggregation feature fusion unit in the template diffusion network, to obtain sample template aggregation feature embeddings; Cross-correlation processing is performed on the sample template aggregation feature embeddings and the sample tracking feature embeddings by a cross-correlation processing unit in the template diffusion network, to obtain sample similarities between each sub-sample region in the sample tracking region and the sample target object; The sample similarities of each sub-sample region are input into a preset loss model to obtain a loss result; and The template diffusion network is trained by the following steps: The template diffusion network is trained by the following steps: Sample initial templates, sample accumulated templates and sample tracking regions are input into the template diffusion network; the sample initial templates include sample target objects; Feature extraction is performed on the sample initial templates, the sample accumulated templates and the sample tracking regions by a feature extraction network in the template diffusion network, to correspondingly obtain sample initial feature embeddings, sample accumulated feature embeddings and sample tracking feature embeddings; Spatial feature aggregation processing is performed on the sample initial feature embeddings and the sample accumulated feature embeddings by a spatial feature aggregation network in the template diffusion network, to obtain sample spatial aggregation features; Channel feature aggregation processing is performed on the sample initial feature embeddings and the sample accumulated feature embeddings by a channel feature aggregation network in the template diffusion network, to obtain sample channel aggregation features; The sample spatial aggregation features and the sample channel aggregation features are fused by an aggregation feature fusion unit in the template diffusion network, to obtain sample template aggregation feature embeddings; Cross-correlation processing is performed on the sample template aggregation feature embeddings and the sample tracking feature embeddings by a cross-correlation processing unit in the template diffusion network, to obtain sample similarities between each sub-sample region in the sample tracking region and the sample target object; The sample similarities of each sub-sample region are input into a preset loss model to obtain a loss result; and According to the loss result, parameters in the feature extraction network, the spatial feature aggregation network and the channel feature aggregation network are corrected to obtain a corrected template propagation network.

10. A target object tracking apparatus characterized by comprising: The method comprises the steps of: The receiving module is configured to receive a target tracking request. The target tracking request comprises an initial template, a current accumulated template and a to-be-tracked image, and the initial template comprises the target object. The feature extraction module is configured to, in response to the target tracking request, perform feature extraction on a to-be-tracked region corresponding to the initial template, the current accumulated template and the to-be-tracked image respectively, and obtain initial feature embedding, current accumulated feature embedding and to-be-tracked feature embedding correspondingly. The to-be-tracked region comprises at least two sub-regions. The aggregation processing module is configured to perform spatial feature aggregation processing on the initial feature embedding and the current accumulated feature embedding to obtain spatial aggregated features, perform channel feature aggregation processing on the initial feature embedding and the current accumulated feature embedding to obtain channel aggregated features, sum elements at each corresponding position of the initial feature embedding, the spatial aggregated features and the channel aggregated features to obtain element sums at the corresponding positions, and determine a template aggregated feature embedding according to the element sums at all positions. The cross-correlation processing module is configured to determine a sub-feature embedding corresponding to each of the sub-regions in the to-be-tracked feature embedding, and determine a similarity between each of the sub-regions and the target object by sliding an aggregated template in the to-be-tracked region.

11. A target object tracking device, comprising: The initial template and the current accumulated template are aggregated to form the aggregated template. The determining module is configured to determine a tracking result of the target tracking request in the at least two sub-regions according to the similarity, and output the tracking result. The method comprises the steps of:

12. A computer-readable storage medium, characterized in that, The memory is configured to store executable instructions. The processor is configured to execute the executable instructions stored in the memory to implement the target object tracking method of any one of claims 1 to 9. The executable instructions are stored in the memory and are configured to cause the processor to execute the executable instructions to implement the target object tracking method of any one of claims 1 to 9.

Images