Target tracking method and apparatus

Abstract

This application discloses a target tracking method and apparatus, belonging to the field of computer vision technology. The target tracking method includes: acquiring template feature information from template frame event data and search feature information from search frame event data; wherein the template frame event data is at least the previous frame of data in the search frame event data, and the template frame event data contains location region information of the target object. After obtaining the template feature information from the template frame event data and the search feature information from the search frame event data, a codec network module combines the template feature information and the search feature information to determine target feature information. The target feature information is then input into a regression network module to obtain the location region information of the target object in the search frame event data. Compared with traditional methods for processing color images, this process has a lower computational load and can improve the processing speed of target tracking.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a target tracking method and apparatus. Background Technology

[0002] With the development of computer vision theory and technology, target tracking has become one of the hot topics in current computer vision research, and it has been widely used in fields such as video surveillance, virtual reality, human-computer interaction, and autonomous driving.

[0003] Currently, target tracking typically relies on analyzing and processing video or image sequences acquired by color vision sensors. For example, based on the size and position of the target object in the first image frame of a video, the target object is tracked in the remaining image frames of the video. However, analyzing and processing image frames in video acquired by color vision sensors is computationally intensive and slow, which cannot meet the speed requirements of target tracking tasks. Summary of the Invention

[0004] This application provides a target tracking method and apparatus to improve the processing speed of target tracking.

[0005] In a first aspect, embodiments of this application provide a target tracking method, which may include: acquiring template feature information of template frame event data and search feature information of search frame event data respectively; wherein the template frame event data is at least the previous frame data of the search frame event data, and the template frame event data contains location region information of the target object. After obtaining the template feature information of the template frame event data and the search feature information of the search frame event data, a codec network module combines the template feature information and the search feature information to determine target feature information, and inputs the target feature information into a regression network module to obtain the location region information of the target object in the search frame event data.

[0006] The target tracking method provided in this application, by processing event data and combining template feature information from template frame event data with search feature information from search frame event data, can determine the location region information of the target object in the search frame event data based on the location region information of the target object contained in the template frame event data. Compared with traditional methods for processing color images, this process involves less computation and can improve the processing speed of target tracking.

[0007] In one possible implementation, for the acquired template frame event data and search frame event data, a first feature extraction module can extract features from the template frame event data to obtain template feature information, and a second feature extraction module can extract features from the search frame event data to obtain search feature information. For example, the first and second feature extraction modules can share network parameters.

[0008] In one possible implementation, the encoder-decoder network module can employ a deep self-attention transform network.

[0009] For example, in one embodiment, the codec network module may include a motion perception module, a target perception module, and a first feature fusion module. After obtaining the template feature information of the template frame event data and the search feature information of the search frame event data, the motion perception module can extract the motion feature information of the moving object from the search feature information. Correlation operations are performed on the template feature information and the search feature information to obtain response feature information, and the target response feature information of the target object is extracted from the response feature information by the target perception module. The first feature fusion module performs feature fusion on the motion feature information and the target response feature information to obtain the target feature information. For example, the first feature fusion module determines similar feature information between the motion feature information and the target response feature information, and performs weighted fusion of the similar feature information and the motion feature information to obtain the target feature information.

[0010] The aforementioned target tracking method uses a motion perception module to identify moving objects in the search frame event data, obtaining motion feature information. It also uses a target perception module to identify target objects in the search frame event data based on template frame event data, obtaining target response feature information. Finally, a feature fusion module fuses the motion feature information and the target response feature information to obtain the target feature information. Based on this target feature information, the regression network module can accurately determine the location region of the target object in the search frame event data, enabling accurate target tracking.

[0011] In another embodiment, the encoding / decoding network module may include an event association module, an object motion description module, and a second feature fusion module. After obtaining the template feature information of the template frame event data and the search feature information of the search frame event data, the template feature information and the search feature information can be correlated to obtain the target response information. The event association module associates the local features and global features contained in the target response information to obtain the target response features. The object motion description module extracts the object motion features from the target response features. The second feature fusion module fuses the target response features and the object motion features to obtain the target feature information.

[0012] The aforementioned target tracking method, through an event association module, associates local and global features contained in the target response information, enhancing the correlation between local and global features and achieving a more detailed description of the target response information, resulting in more accurate target response features. Then, an object motion description module extracts object motion features from the target response features, and a second feature fusion module fuses these features to obtain target feature information. Moving objects in the search frame event data are identified to obtain motion feature information. Furthermore, a target perception module, based on template frame event data, identifies target objects in the search frame event data to obtain target response feature information, and the feature fusion module fuses the motion feature information and target response feature information to obtain target feature information. Based on the target feature information obtained in the above manner, the regression network module can accurately determine the location region information of the target object in the search frame event data, enabling accurate target tracking.

[0013] In one possible implementation, both the template frame event data and the search frame event data are event data acquired by a dynamic vision sensor. The template frame event data can be the event data of the previous n frames preceding the search frame event data, where n can be a positive integer. The location region information of the target object in the template frame event data can be determined by a target detection algorithm or a target tracking method. For example, for the first frame of event data acquired by the dynamic vision sensor, a target detection algorithm can be used to determine the location region information of the target object contained therein; for the second frame of event data acquired by the dynamic vision sensor, the location region information of the target object in the second frame of event data can be determined based on the location region information of the target object in the first frame of event data, using the target tracking method provided in the embodiments of this application; for the third frame of event data acquired by the dynamic vision sensor, the location region information of the target object in the third frame of event data can be determined based on the location region information of the target object in the first or second frame of event data.

[0014] Secondly, embodiments of this application also provide a target tracking device, including: a memory and a processor; wherein, the memory is used to store computer instructions; and the processor is used to read the computer instructions from the memory and perform the following steps:

[0015] The template feature information of the template frame event data and the search feature information of the search frame event data are obtained respectively; wherein, the template frame event data is at least the previous frame data of the search frame event data, and the template frame event data contains the location region information of the target object;

[0016] The target feature information is determined by combining template feature information and search feature information through the encoding and decoding network module;

[0017] The target feature information is input into the regression network module to obtain the location region information of the target object in the search frame event data.

[0018] In one possible implementation, the encoder-decoder network module can employ a deep self-attention transform network.

[0019] In one possible implementation, the codec network module includes a motion perception module, a target perception module, and a first feature fusion module; the processor can specifically be used to: extract motion feature information of a moving object from search feature information through the motion perception module; perform correlation operations on template feature information and search feature information to obtain response feature information, and extract target response feature information of the target object from the response feature information through the target perception module; and perform feature fusion on the motion feature information and the target response feature information through the first feature fusion module to obtain target feature information.

[0020] In one possible implementation, the processor can be specifically used to: determine similar feature information between motion feature information and target response feature information through the first feature fusion module, and perform weighted fusion of similar feature information and motion feature information to obtain target feature information.

[0021] In one possible implementation, the codec network module may include an event association module, an object motion description module, and a second feature fusion module; the processor may specifically be used to: perform correlation operations on template feature information and search feature information to obtain target response information; associate local features and global features contained in the target response information through the event association module to obtain target response features; extract object motion features from the target response features through the object motion description module; and fuse the target response features and object motion features through the second feature fusion module to obtain target feature information.

[0022] Thirdly, this application provides a computer-readable storage medium storing a computer program or instructions, which, when executed by a terminal device, causes the terminal device to perform any of the methods provided in the first aspect above.

[0023] Fourthly, this application provides a computer program product comprising a computer program or instructions that, when executed by a terminal device, implement any of the methods provided in the first aspect or the second aspect.

[0024] The technical effects that can be achieved by any of the second to fourth aspects mentioned above can be referred to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] Figure 1 A schematic diagram of an event flow generated by a dynamic vision sensor;

[0026] Figure 2 A flowchart illustrating a target tracking method provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the data flow of a target tracking method provided in an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the structure of a neural network model for target tracking provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the network structure of a motion sensing module provided in an embodiment of this application;

[0030] Figure 6 A schematic diagram of the network structure of a target perception module provided in an embodiment of this application;

[0031] Figure 7 A schematic diagram of the network structure of a first feature fusion module provided in an embodiment of this application;

[0032] Figure 8 A schematic diagram of another neural network model for target tracking provided in this application embodiment;

[0033] Figure 9 A schematic diagram of the network structure of an event association module provided in an embodiment of this application;

[0034] Figure 10 A schematic diagram of the network structure of an object motion description module provided in an embodiment of this application;

[0035] Figure 11 A schematic diagram of the network structure of a second feature fusion module provided in an embodiment of this application;

[0036] Figure 12 This application provides a schematic diagram of the network structure of a regression network module.

[0037] Figure 13 A comparison diagram showing the effect of the target tracking method provided in this application embodiment with other methods;

[0038] Figure 14 A schematic diagram illustrating an example of a target tracking device provided in an embodiment of this application;

[0039] Figure 15 This is a schematic diagram of another example of a target tracking device provided in the embodiments of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only used to explain specific embodiments of this application and is not intended to limit this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] Before introducing the specific solutions provided in the embodiments of this application, some terms used in this application will be explained in a general way to facilitate understanding by those skilled in the art, and the terms used in this application will not be limited.

[0042] (1) Color vision sensor: also known as an RGB sensor, where RGB is an abbreviation for red, green, and blue. RGB cameras are used to capture still images or videos and typically include a lens and a photosensitive element. An object passes through the lens to generate an optical image that is projected onto the photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an image signal processor. The image signal processor can convert the received electrical signal into a digital image signal and further obtain an image or video.

[0043] (2) Dynamic Vision Sensor (DVS): This is a biomimetic sensor, also known as an event camera, event-driven camera, or event camera sensor. It is a new type of camera that has emerged in recent years. Unlike traditional RGB cameras that capture complete images, event cameras capture "events." An event camera treats the brightness change of a single pixel in a real-world scene as an event. Event cameras capture dynamic changes in a scene based on an event-driven approach. When objects in a real-world scene change, the event camera generates a spatiotemporal data stream of a series of events, called event data.

[0044] In this application embodiment, "multiple" refers to two or more. Therefore, in this application embodiment, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, such as one, two, or more. For example, "including at least one" means including one, two, or more, and it does not limit which ones are included. For example, including at least one of A, B, and C, then it could include A, B, C, A and B, A and C, B and C, or A and B and C. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0045] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects, and are not used to limit the order, sequence, priority or importance of multiple objects.

[0046] Currently, target tracking methods track targets by analyzing and processing image frames in video captured by color vision sensors, which is slow. For example, a color vision sensor captures video at a rate of 60 frames per second, while the target tracking method can only process 5 image frames per second to determine the target object's position within those frames. Clearly, the processing speed of target tracking methods lags far behind the video speed, failing to meet the speed requirements of target tracking tasks.

[0047] Based on this, embodiments of this application provide a target tracking method. This method is applicable to a target tracking device capable of acquiring and processing event data, which can be understood as an electronic device. For example, the target tracking device can have a built-in or external dynamic vision sensor to acquire event data collected by the dynamic vision sensor; or, the target tracking device can connect to other devices storing event data via wired or wireless connection to receive event data transmitted from other devices and process the event data.

[0048] For example, the target tracking device can be a monitoring device fixedly installed in a certain location, or a back-end server of a monitoring system; the back-end server can receive event data remotely transmitted by dynamic visual sensors via a network and process the event data. The target tracking device can also be a device that provides users with shooting functions or data connectivity, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem, such as smartphones, portable, pocket-sized, handheld, wearable devices, tablets, personal computers (PCs), PDAs (Personal Digital Assistants), in-vehicle computers, drones, aerial photography drones, computers, etc.

[0049] The working principle of a dynamic vision sensor will be explained below. A dynamic vision sensor can capture changes in light intensity within a scene and generate event stream data based on these changes. Specifically, when the light intensity of a pixel in the scene changes, the dynamic vision sensor generates an event. For example... Figure 1 As shown, the data corresponding to an event can include (t,<x,y> It consists of four parts: t, p, and t, where t is the timestamp of the triggering event, i.e., the time when the brightness of the pixel changes.<x,y> The coordinates of the triggering event are the row and column values ​​of the light sensor corresponding to the pixel whose light intensity changed; p represents the polarity of the event, indicating whether the light intensity change is an increase or a decrease. When acquiring the pixel's light intensity data, a logarithmic function can be used to convert the pixel's light intensity signal into an electrical signal, thereby obtaining the pixel's light intensity data and determining the change in light intensity. For example, the polarity p can be expressed as... Where I(x,y) represents the coordinate position.<x,y> Light intensity data of the pixel at that location, Indicates coordinate position<x,y> The dynamic vision sensor measures the change in light intensity data at a given pixel. When the change in light intensity data exceeds a set positive threshold, p can be 1, indicating an increase in the light intensity of that pixel. When the change in light intensity data falls below a set negative threshold, p can be -1, indicating a decrease in the light intensity of that pixel. Because the dynamic vision sensor generates events only at locations with changes in light intensity, it provides very high temporal resolution and very low power consumption. Furthermore, since the change in light intensity is calculated on a logarithmic scale, the dynamic vision sensor can operate within a very high dynamic range, i.e., a wide range of light intensity variations.

[0050] When an object moves in a scene, the light intensity at the corresponding location changes, and the dynamic vision sensor generates a series of event streams. The dynamic vision sensor accumulates these event streams according to a set time period, outputting event data at a fixed frame rate. For example, the set time period could be 5ms, meaning that the event stream within each 5ms frame is accumulated to generate one frame of event data. Therefore, the event data output by the dynamic vision sensor consists of a sequence of multiple event data points within a certain time period.

[0051] The target tracking device provided in this application embodiment can perform target tracking based on event data collected by a dynamic vision sensor. For example, the target tracking device can obtain template frame event data and search frame event data from the event data, wherein the template frame event data is at least the previous frame of the search frame event data, and the template frame event data is marked with the location region information of the target object. For example, the template frame event data can be event data whose location region of the target object has been marked by a target detection method or a target tracking method. The target tracking device can obtain template feature information from the template frame event data and search feature information from the search frame event data. Using an encoding / decoding network module, it combines the template feature information and the search feature information to determine the target feature information, and inputs the target feature information into a regression network module to obtain the location region of the target object in the search frame event data. This application embodiment processes event data using the above method, with low computational load, thus improving the processing speed of target tracking.

[0052] Figure 2 A flowchart illustrating a target tracking method provided in an embodiment of this application is shown. Figure 2 As shown, the method may include the following steps:

[0053] S201, respectively obtain the template feature information of the template frame event data and the search feature information of the search frame event data.

[0054] The template frame event data is at least the previous frame of the search frame event data, and the template frame event data contains the location region information of the target object.

[0055] In some embodiments, the target tracking device can acquire event data collected by a dynamic vision sensor via a wired or wireless connection. For the first frame of event data, target detection technology can be used to detect the location region of the target object in the first frame of event data, and the location region information of the target object can be marked in the first frame of event data. The target object can be a specified person, an animal, or a moving object, etc., and this application does not limit this. After marking the location region information of the target object in the first frame of event data, for any subsequent frame of event data, target detection technology or the target tracking method provided in the embodiments of this application can be used to determine the location region information of the target object in the corresponding frame of event data. For example, assuming that the location region information of the target object has been marked in the (N-1)th frame of event data, the location region information of the target object can be determined in the Nth frame of event data or the (N+n)th frame of event data based on the location region information of the target object in the (N-1)th frame of event data, using the target tracking method provided in the embodiments of this application, where N is an integer greater than 1 and n is an integer greater than or equal to 1. The (N-1)th frame of event data is the frame preceding the Nth frame of event data.

[0056] It's important to note that while object detection technology for finding target objects in event data requires comparing each feature point in the event data with the feature points of the target object one by one, this process is time-consuming. In contrast, object tracking methods, which determine the target object's location in the next frame of event data based on the previous frame's location information, require significantly less time than object detection. Therefore, for most frames of event data where the scene remains relatively unchanged, object tracking methods can be used to process the event data and determine the target object's location, saving time.

[0057] For example, when it is necessary to determine the location region information of the target object in the Nth frame of event data, the Nth frame of event data can be used as the search frame of event data, and the (N-1)th frame of event data can be used as the template frame of event data. Features can be extracted from the template frame of event data and the search frame of event data using convolutional neural networks (CNNs) respectively, to obtain the template feature information of the template frame of event data and the search feature information of the search frame of event data.

[0058] S202 uses an encoding / decoding network module to determine target feature information by combining template feature information and search feature information.

[0059] The encoding / decoding network module can employ a deep self-attention transform network, such as a transformer network structure. Transformer networks can compute representations of inputs and outputs by relying on a self-attention mechanism.

[0060] In some embodiments, the codec network module may include a motion perception module, a target perception module, and a first feature fusion module. The motion perception module extracts motion features of the moving object from the search feature information; performs correlation operations on the template feature information and the search feature information to obtain response feature information; and the target perception module extracts the target response feature information (target embedding) of the target object from the response feature information. Finally, the first feature fusion module fuses the motion feature information and the target response feature information to obtain the target feature information.

[0061] In other embodiments, the encoding / decoding network module may include an event association module, an object motion description module, and a second feature fusion module. The template feature information and search feature information are correlated to obtain target response information. The event association module associates the local and global features contained in the target response information to obtain target response features. The object motion description module extracts object motion features from the target response features. The second feature fusion module fuses the target response features and object motion features to obtain target feature information.

[0062] S203, Input the target feature information into the regression network module to obtain the location region information of the target object in the search frame event data.

[0063] The regression network module can employ a deep learning-based regression network. This network outputs bounding boxes based on target feature information, identifying the location of the target object within the search frame event data. The bounding box is a rectangular box that encloses the target object.

[0064] For example, such as Figure 3 As shown, in this embodiment, the acquired template frame event data and search frame event data are input into a convolutional neural network. The convolutional neural network extracts features from the template frame event data and search frame event data respectively to obtain template feature information and search feature information. An encoding / decoding network module is used to encode and decode the obtained template feature information and search feature information to obtain target feature information. The target feature information is then input into a regression network module to obtain the target bounding box output by the regression network module, thereby determining the location region information of the target object in the search frame event data.

[0065] The embodiments of this application process event data using the above method, which requires little computation and thus improves the processing speed of target tracking. Furthermore, by employing a codec network module and combining the template feature information of the template frame event data with the search feature information of the search frame event data, the location region of the target object in the search frame event data can be determined, enabling accurate tracking of the target object.

[0066] For example, the dynamic vision sensor accumulates the collected event stream according to a set time period of 5ms and outputs event data frames with a fixed frame rate of 200 frames / second. Using the target tracking method provided in this application embodiment, the processing time for each frame of event data is very short, thus meeting the speed of the dynamic vision sensor outputting event data and the processing speed requirements of the target tracking task.

[0067] For ease of understanding, the target tracking method provided in this application embodiment will be described in detail below through two specific examples.

[0068] In some embodiments, such as Figure 4 As shown, the encoding / decoding network module may include a target perception module 403, a motion perception module 404, and a first feature fusion module 405. That is, the target perception module 403, the motion perception module 404, and the first feature fusion module 405 together form a transformer encoding / decoding network.

[0069] After acquiring template frame event data and search frame event data, the template frame event data can be input into the first feature extraction module 401, which extracts features from the template frame event data to obtain template feature information. The search frame event data can be input into the second feature extraction module 402, which extracts features from the search frame event data to obtain search feature information. Both the first and second feature extraction modules can employ neural networks for feature extraction, such as CNN networks or backbone networks. In some embodiments, the first feature extraction module 401 and the second feature extraction module 402 can share network parameters.

[0070] Both template frame event data and search frame event data can be understood as event frames formed by accumulating the polarity of events at each pixel within a short time range. The sizes of template frame event data and search frame event data can be the same or different. For example, the size of template frame event data can be 127*127, and the size of search frame event data can be 303*303.

[0071] Considering that event data contains not only contour information but also object motion information, the motion feature information of the moving object can be extracted from the search feature information of the search frame event data by the motion sensing module 404. The search feature information of the search frame event data can be understood as the global feature information of the search frame event data. The motion sensing module 404 can also be called a motion-aware network and can adopt a transformer encoder structure. Since the search frame event data used in this embodiment is accumulated according to the polarity of the trigger events within a certain time, the motion pattern is different for different moving objects and backgrounds. Therefore, the transformer encoder structure can be used to filter out the background features from the global feature information of the search frame event data and retain the features of the moving object.

[0072] Figure 5 An exemplary network structure diagram of a motion sensing module provided in an embodiment of this application is shown. Figure 5 As shown, the motion perception module 404 may include a first multi-head self-attention network and a first feed-forward neural network (FFN). Summation and normalization (add & normal) layers are set before and after the first feed-forward neural network. Position encoding information and search feature information from search frame event data are summed to obtain search-related feature information, where the position encoding information is a pre-defined fixed encoding sequence. Both the search-related feature information and the search feature information are used as inputs to the first multi-head self-attention network. The first multi-head self-attention network processes the search-related feature information and the search feature information. Then, the summation and normalization layers connected to the first multi-head self-attention network perform residual connection and normalization processing on the output of the first multi-head self-attention network and the search feature information to obtain search attention feature information. The residual connection is used to prevent network degradation. The search attention feature information is input into the first feedforward neural network. The first feedforward neural network processes the search attention feature information to obtain the features of the moving object output by the first feedforward neural network. Then, the search attention feature information and the features of the moving object output by the first feedforward neural network are subjected to residual connection and normalization processing through the summation and normalization layer connected to the first feedforward neural network to obtain the motion feature information of the search frame event data.

[0073] To improve the ability to identify target objects, embodiments of this application utilize a target perception module 403 to obtain the feature information of the target object. The target perception module 403 can be referred to as a target-aware network or can employ a transformer encoder structure. In some embodiments, the template feature information of the template frame event data and the search feature information of the search frame event data are first processed through a correlation operation to obtain initial response feature information, also known as a feature response map. This response feature information is then input into the target perception module 403, which can extract the target response feature information of the target object from the template feature information and filter out the response features of the target object from other non-template feature information.

[0074] Figure 6 An exemplary network structure diagram of a target perception module provided in an embodiment of this application is shown. Figure 6 As shown, the target perception module 403 may include three multi-head self-attention networks and two feedforward neural networks. The three multi-head self-attention networks are a second multi-head self-attention network, a third multi-head self-attention network, and a fourth multi-head self-attention network, respectively. The two feedforward neural networks are a second feedforward neural network and a third feedforward neural network, respectively. Each multi-head self-attention network and each feedforward neural network is followed by a summation and normalization layer.

[0075] After performing correlation operations on template feature information and search feature information, response feature information is obtained. The response feature information and position encoding information are then summed to obtain response-related feature information. Both response-related feature information and response feature information are used as inputs to a second multi-head self-attention network (MNB). The MNB processes the response-related feature information and response feature information, and then performs residual connection and normalization processing on the output of the second MNB and the response feature information through a summation and normalization layer connected to the second MNB to obtain response attention feature information. This response attention feature information is then input to a second feedforward neural network (FPNN). The second FNB processes the response attention feature information to obtain the response attention feature output by the second FNB. This response attention feature is then performed residual connection and normalization processing on the response attention feature information and the response attention feature output by the second FNB through a summation and normalization layer connected to the second FNB to obtain the target response feature. Finally, the target response feature is summed with the position encoding information to obtain the target position encoding information.

[0076] The pre-defined target sequence (target query) and its summation result are input into a third multi-head self-attention network. Then, a summation and normalization layer connected to the third multi-head self-attention network performs residual connection and normalization processing on the output and response feature information of the third multi-head self-attention network to obtain the first sequence feature information. The first sequence feature information is summed with the target sequence to obtain the second sequence feature information. The target response feature, target position encoding information, and second sequence feature information are input into a fourth multi-head self-attention network. The fourth multi-head self-attention network processes these features, and a summation and normalization layer connected to it performs residual connection and normalization processing on the output of the fourth multi-head self-attention network and the first sequence feature information to obtain the target attention feature information. The target attention feature information is then input into a third feedforward neural network. The third feedforward neural network processes this feature information, and a summation and normalization layer connected to it performs residual connection and normalization processing on the target attention feature information and the output of the third feedforward neural network to obtain the target response feature information.

[0077] The target response feature information output by the target perception module 403 and the motion feature information output by the motion perception module 404 are input into the first feature fusion module 405. The first feature fusion module 405 performs feature fusion on the motion feature information and the target response feature information to obtain the target feature information. The first feature fusion module 405 can be called the target fusion module and can adopt a transformer decoder structure. The first feature fusion module 405 can calculate the similarity between the motion feature information and the target response feature information and then weightedly fuse them into the motion feature information to obtain the target feature information, which can also be called the target response map.

[0078] Figure 7 An exemplary network structure diagram of a first feature fusion module provided in an embodiment of this application is shown. Figure 7 As shown, the first feature fusion module 405 may include a dot product unit, an element-wise product unit, and multiple convolutional layers. The dot product unit is used to multiply the lengths of two vectors and then multiply by the cosine of the angle between the two vectors to obtain the dot product of the two vectors. The element-wise product unit is used to multiply the two vectors element by element to form a new result vector. The result vector has the same size as the original vector and can be regarded as an element-wise product of the two vectors.

[0079] Motion feature information is passed through the first convolutional layer to obtain the first convolution result, and target response feature information is passed through the second convolutional layer to obtain the second convolution result. Both the first and second convolution results can be understood as vectors. A dot product unit performs a dot product between the first convolution result corresponding to the motion feature information and the second convolution result corresponding to the target response feature information to obtain similarity feature information between the two. This similarity feature information, as the dot product result, can also be understood as a vector. An element-wise product unit determines the element-wise product of the first convolution results corresponding to the similarity feature information and the motion feature information, obtaining the element-wise product result. This element-wise product result is then passed through the third convolutional layer to obtain the target feature information.

[0080] The target feature information is input into the regression network module 406 to obtain the target bounding box output by the regression network module 406. The target bounding box is used to identify the location region information of the target object in the search frame event data. The regression network module 406 may include multiple convolutional layers and network layers for implementing the soft-argmax function. The target feature information first passes through multiple convolutional layers, and then through the soft-argmax function to finally obtain the location information of the target bounding box.

[0081] The above embodiments combine the feature extraction and fusion capabilities of CNN networks with the global feature information processing capabilities of transformer networks to provide a new network structure. This network structure uses a motion perception module suitable for event data to identify moving objects in the search frame event data and obtain motion feature information. It also uses a target perception module suitable for event data to identify target objects in the search frame event data based on template frame event data and obtain target response feature information. Finally, a feature fusion module fuses the motion feature information and the target response feature information, thereby accurately determining the location region information of the target object in the search frame event data and accurately tracking the target object.

[0082] The encoding / decoding network module and regression network module used in the above embodiments can be trained together. The training process may include the following steps: Acquiring training sample event data, which may include multiple pairs of template frame sample data and search frame sample data. The template frame sample data contains bounding boxes, which are used to identify the location region of the target object within the template frame sample data. The search frame sample data contains annotation information, which indicates the position of the target object within the search frame sample data. The paired template frame sample data and search frame sample data are respectively input into the trained first feature extraction module and second feature extraction module to obtain template sample feature information and search sample feature information. The search sample feature information is input into the motion perception module to be trained to obtain sample motion feature information output by the motion perception module. Correlation operations are performed on the template sample feature information and search sample feature information to obtain sample response feature information. The sample response feature information is input into the target perception module to be trained to obtain sample target response feature information output by the target perception module. The sample motion feature information and sample target response feature information are input into the first feature fusion module to be trained to obtain sample target feature information output by the first feature fusion module. The target feature information of the sample is input into the regression network module to be trained, and the target box annotation result output by the regression network module is obtained. Based on the target box annotation result and the annotation information corresponding to the search frame sample data, the training loss value is determined, and the network parameters of the encoding / decoding network module and the regression network module are adjusted according to the loss value. The next round of training is carried out until the obtained loss value meets the set expectation value, and the training is stopped, and the trained encoding / decoding network module and regression network module are obtained.

[0083] In other embodiments, such as Figure 8 As shown, the encoding / decoding network module may include an event association module 803, an object motion description module 804, and a second feature fusion module 805. That is, the event association module 803, the object motion description module 804, and the second feature fusion module 805 together form the transformer encoding / decoding network.

[0084] After acquiring the template frame event data and the search frame event data, the template frame event data can be input into the first feature extraction module 801 to obtain template feature information, and the search frame event data can be input into the second feature extraction module 802 to obtain search feature information. The first and second feature extraction modules constitute an event-based feature extraction module, used to extract features from the template frame event data and the search frame event data. The event-based feature extraction module can be understood as a convolutional neural network based on a Siamese network structure. The first feature extraction module is the template branch of the event-based feature extraction module, and the second feature extraction module is the search branch of the event-based feature extraction module. The first and second feature extraction modules can share the same architecture. For example, the first and second feature extraction modules can employ a residual neural network, ResNet18.

[0085] The similarity between the template feature information output by the first feature extraction module 801 and the search feature information output by the second feature extraction module 802 is determined using relevant algorithms and output as a response feature map, which can serve as the target response information. If the target response information is directly regressed to determine the location of the target bounding box and track the target object, the relationship between local and global features cannot be well modeled. Considering that local features contain compact vector representations of the local image neighborhood, while global features contain long-range contour and shape representations, this embodiment inputs the target response information into the event association module 803. The event association module 803 associates the local and global features contained in the target response information to obtain the target response features.

[0086] Figure 9 An exemplary network structure diagram of an event association module provided in an embodiment of this application is shown. Figure 9As shown, the event correlation module 803 may include a fifth multi-head self-attention network and a fourth feedforward neural network. Summation and normalization layers are set before and after the fourth feedforward neural network. The event correlation module 803 can also be called a local-global event correlation module. The target response information is used as matrix V1. The target response information and a portion of the spatial positional encoding information are summed to obtain matrix K1. The target response information and another portion of the spatial positional encoding information are summed to obtain matrix Q1. The spatial positional encoding information is a pre-defined fixed encoding sequence. Matrices K1 and Q1 are used to represent the local features contained in the target response information, while matrix V1 represents the global features contained in the target response information. Matrix V1, matrix K1, and matrix Q1 are input into a fifth multi-head self-attention network. The fifth multi-head self-attention network processes these matrices, and then a summation and normalization layer connected to the fifth multi-head self-attention network performs residual connection and normalization processing on the output of the fifth multi-head self-attention network and the target response information to obtain event-related feature information. This event-related feature information is then input into a fourth feedforward neural network. The fourth feedforward neural network processes this information, and then a summation and normalization layer connected to the fourth feedforward neural network performs residual connection and normalization processing on the event-related feature information and the output of the fourth feedforward neural network to obtain target response features. The event association module 803 enhances the local-global correlation of objects and their context regions, thereby achieving a more detailed description of the target response information.

[0087] To better optimize the obtained target response features, this application embodiment processes the target response features through an object motion description module, which can also be called an object motion descriptor. It can use the object query and transformer module to learn the object motion features and then extract the object motion features from the target response features. Figure 10 An exemplary diagram illustrates a network structure of an object motion description module provided in an embodiment of this application. For example... Figure 10As shown, the object motion description module 804 may include a sixth multi-head self-attention network, a seventh multi-head self-attention network, and a fifth feedforward neural network. Each of these networks is followed by a summation and normalization layer. A pre-set target sequence is used as matrix V2. A portion of the target sequence is copied, and the target sequence and the copied portion are summed to obtain matrix K2. Another portion of the target sequence is copied, and the target sequence and the copied portion are summed to obtain matrix Q2. Matrix V2, matrix K2, and matrix Q2 are input into the sixth multi-head self-attention network. The network processes these matrices, and then the output of the sixth multi-head self-attention network and the target sequence are residually connected and normalized through the summation and normalization layer connected to the network to obtain target sequence feature information. The target response features are used as matrix V3. The target response features and spatial position encoding information are summed to obtain matrix K3. The target sequence feature information and the target sequence are summed to obtain matrix Q3. Matrix V3, matrix K3, and matrix Q3 are input into the seventh multi-head self-attention network. The seventh multi-head self-attention network processes these matrices. Then, a summation and normalization layer connected to the seventh multi-head self-attention network performs residual connection and normalization processing on the output of the seventh multi-head self-attention network and the target sequence feature information to obtain the object motion feature information. This object motion feature information is then input into the fifth feedforward neural network. The fifth feedforward neural network processes this information, and a summation and normalization layer connected to the fifth feedforward neural network performs residual connection and normalization processing on the object motion feature information and the output of the fifth feedforward neural network to obtain the object motion features.

[0088] The target response features output by the event association module 803 and the object motion features output by the object motion description module 804 are input into the second feature fusion module. The second feature fusion module performs feature fusion on the target response features and the object motion features to obtain target feature information.

[0089] Figure 11 An exemplary diagram illustrates a network structure of a second feature fusion module provided in an embodiment of this application. The second feature fusion module can also be referred to as an object bounding box regressor. Figure 11As shown, the second feature fusion module 805 may include a dot product unit, an element-wise product unit, and multiple convolutional layers. The target response feature passes through a third convolutional layer to obtain a third convolution result, and the object motion feature passes through a fourth convolutional layer to obtain a fourth convolution result. Both the third and fourth convolution results can be understood as vectors. The dot product unit performs a dot product between the third convolution result corresponding to the target response feature and the fourth convolution result corresponding to the object motion feature, obtaining a dot product result, which can also be understood as a vector. The element-wise product unit determines the element-wise product between the dot product result and the third convolution result corresponding to the target response feature, obtaining an element-wise product result. The element-wise product result passes through a sixth convolutional layer to obtain the target feature information.

[0090] The target feature information is input into the regression network module, which then processes the target feature information. Figure 12 An exemplary diagram illustrates a network structure of a regression network module provided in an embodiment of this application. For example... Figure 12 As shown, the regression network module 806 may include fully convolutional networks (FCNs) and two network branches connected to the FCNs. Each network branch includes a convolutional layer and a network layer for implementing the soft-argmax function. The FCNs may include convolutional layers and pooling layers, which can be alternated. The FCNs can output a segmentation map of the same size as the input map. The segmentation map contains not only semantic information of each object but also positional information of each object. The semantic information is used to determine the category of the object.

[0091] In this embodiment, target feature information can be understood as a feature map containing feature information of the target object. The feature map is input into a fully convolutional network, which outputs a segmentation map of the same size as the feature map. The segmentation map includes the location and semantic information of the target object. The segmentation map is then input into each network branch for regression processing. The first network branch outputs the coordinates of the upper left corner of the target box based on the location and semantic information of the target object in the segmentation map. The second network branch outputs the coordinates of the lower right corner of the target box based on the location and semantic information of the target object in the segmentation map. Therefore, the regression network module 806 can output a target box, which is used to identify the location region information of the target object in the search frame event data.

[0092] The target tracking method provided in this application embodiment can determine the location region information of a target object in search frame event data based on template frame event data containing the target object's location region information. The template frame event data can be the previous n frames of event data preceding the search frame event data, where n is an integer greater than or equal to 1. This application embodiment combines a CNN network and a transformer network to process the event data, extracting the target object's feature information. This allows the regression network module to directly output regression results based on the target object's feature information without complex post-regression processing, thus obtaining the target object's location region information. This not only reduces computational load but also improves processing speed.

[0093] The encoding / decoding network module and regression network module used in the above embodiments can be trained together. The training process may include the following steps: Acquiring training sample event data, which may include multiple pairs of template frame sample data and search frame sample data. The template frame sample data contains bounding boxes, which are used to identify the location region of the target object within the template frame sample data. The search frame sample data contains annotation information, which indicates the position of the target object within the search frame sample data. The paired template frame sample data and search frame sample data are input into the trained first feature extraction module and second feature extraction module, respectively, to obtain template sample feature information and search sample feature information. Correlation operations are performed on the template sample feature information and search sample feature information to obtain sample target response information. The sample target response information is input into the event association module to be trained to obtain the sample target response features output by the event association module. The sample target response features are input into the object motion description module to be trained to obtain the sample object motion features output by the object motion description module. The sample target response features and sample object motion features are input into the second feature fusion module to be trained to obtain the sample target feature information output by the second feature fusion module. The target feature information of the sample is input into the regression network module to be trained, and the target box annotation result output by the regression network module is obtained. Based on the target box annotation result and the annotation information corresponding to the search frame sample data, the training loss value is determined, and the network parameters of the encoding / decoding network module and the regression network module are adjusted according to the loss value. The next round of training is carried out until the obtained loss value meets the set expectation value, and the training is stopped, and the trained encoding / decoding network module and regression network module are obtained.

[0094] The target tracking method provided in this application is compared with related technologies such as the transformer target tracker and the DimP50 target tracker to determine the intersection-over-union (IOU) ratio between the target bounding boxes marked by various methods and the actual target bounding boxes. Here, the actual target bounding box refers to the actual location region of the target object, pre-marked manually or by other means. The IOU ratio is the ratio of the intersection to the union of the target bounding boxes marked by the tested method and the actual target bounding boxes; it can also be called the overlap rate. Figure 13 As shown in the experimental results, the target tracking method provided in this application has the highest intersection-union ratio (IUR) between the marked target boxes and the actual target boxes, while the IURs of the marked target boxes by the transformer target tracker and the DimP50 target tracker are both very small. Therefore, the target tracking method provided in this application has higher accuracy and stability.

[0095] Based on the same inventive concept as the above embodiments, this application also provides a target tracking device, which can be used to implement the functions of the above method embodiments, and thus can achieve the beneficial effects of the above method embodiments.

[0096] The target tracking device may include a feature extraction unit and a regression unit. The target tracking device is used to implement the above... Figure 2 The function shown in the method embodiment. When the target tracking device is used to implement Figure 2 The function of the method embodiment shown is as follows: the feature extraction unit can be used to execute S201 and S202, and the regression unit can be used to execute S203. For example: the feature extraction unit is used to obtain template feature information of template frame event data and search feature information of search frame event data respectively; through the encoding and decoding network module, the target feature information is determined by combining the template feature information and the search feature information; the template frame event data is at least the previous frame data of the search frame event data, and the template frame event data contains the location region information of the target object; the regression unit is used to input the target feature information into the regression network module to obtain the location region information of the target object in the search frame event data according to the indication information.

[0097] In one possible implementation, the codec network module may employ a deep self-attention transform network.

[0098] In one possible implementation, the encoding / decoding network module includes a motion perception module, a target perception module, and a first feature fusion module; the feature extraction unit can be specifically used to: extract motion feature information of a moving object from search feature information through the motion perception module; perform correlation operations on template feature information and search feature information to obtain response feature information, and extract target response feature information of the target object from the response feature information through the target perception module; and perform feature fusion on motion feature information and target response feature information through the first feature fusion module to obtain target feature information.

[0099] In one possible implementation, the feature extraction unit may be used to: determine similar feature information between motion feature information and target response feature information through the first feature fusion module, and perform weighted fusion of similar feature information and motion feature information to obtain target feature information.

[0100] In one possible implementation, the encoding / decoding network module includes an event association module, an object motion description module, and a second feature fusion module; the feature extraction unit can specifically be used to: perform correlation operations on template feature information and search feature information to obtain target response information; associate local features and global features contained in the target response information through the event association module to obtain target response features; extract object motion features from the target response features through the object motion description module; and fuse the target response features and object motion features through the second feature fusion module to obtain target feature information.

[0101] In one possible implementation, the feature extraction unit can be specifically used to: extract features from template frame event data through a first feature extraction module to obtain template feature information; and extract features from search frame event data through a second feature extraction module to obtain search feature information.

[0102] Based on the same inventive concept as the above embodiments, this application also provides a target tracking device. This target tracking device can be a computing chip, a mobile terminal, a monitoring device, or a dynamic vision sensor, or it can be an electronic device such as a computer or server. The target tracking device can be used to implement the functions of the above method embodiments, and therefore can achieve the beneficial effects of the above method embodiments.

[0103] Figure 14 An exemplary schematic diagram of a target tracking device provided in an embodiment of this application is shown, such as... Figure 14 As shown, the target tracking device 1400 may include a processor 1401 and a memory 1402 connected to the processor 1401.

[0104] Optionally, the processor 1401 and the memory 1402 can be interconnected via a bus. The processor 1401 can be a general-purpose processor, such as a microprocessor, or other conventional processor. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0105] The memory 1402 can be used to store computer-executable program code, including instructions. The memory 1402 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as a target tracking application), etc. The data storage area may store data created during the use of the target tracking device 1400 (such as acquired or collected event data, and the processing results of event data), etc. Furthermore, the memory 1402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0106] In some embodiments, the target tracking device 1400 may further include a data transmission interface, which may be a transceiver. For example, when the target tracking device 1400 is a server, it can receive event data uploaded by a dynamic visual sensor or monitoring device through the transceiver and store the received event data in the memory 1402.

[0107] The processor 1401 executes various functional applications and data processing of the target tracking device 1400 by running instructions stored in the memory 1402 and / or instructions stored in memory disposed within the processor. The memory 1402 may also store corresponding program code for target tracking. When the corresponding program code stored in the memory 1402 is executed by the processor 1401, functions such as... Figure 2 The steps of the target tracking method shown are illustrated. Of course, the relevant event data and program code provided in this embodiment can also be stored in external memory. In this case, the processor 1401 can run the corresponding program code stored in the external memory through the external memory interface to achieve the following: Figure 2 The steps of the target tracking method are shown.

[0108] In other embodiments, the target tracking device may also be a monitoring device that includes a dynamic visual sensor. See also Figure 15As shown, the monitoring device 1500 may include a processor 1501, a memory 1502, a dynamic vision sensor 1503, a transceiver 1504, etc. The processor 1501, memory 1502, dynamic vision sensor 1503, and transceiver 1504 are connected via a bus.

[0109] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the monitoring device 1500. In other embodiments of this application, the monitoring device 1500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0110] Processor 1501 may include one or more processing units, such as an application processor (AP), a modem processor, a controller, a digital signal processor (DSP), a baseband processor, etc. These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution.

[0111] The processor 1501 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1501 is a cache memory. This memory can store instructions or data that the processor 1501 has just used or that are used repeatedly. If the processor 1501 needs to use the instruction or data again, it can directly retrieve it from the cache memory, avoiding repeated accesses, reducing the waiting time of the processor 1501, and thus improving the efficiency of the system.

[0112] In some embodiments, processor 1501 may include one or more interfaces for connection to buses and other modules in monitoring device 1500.

[0113] The dynamic vision sensor 1503 can capture dynamic changes in a scene, generate an event stream, and compress the acquired event stream into frames according to a set time resolution to form event data for output. The monitoring device 1500 processes the event data output by the dynamic vision sensor 1503 through the processor 1501 to achieve target tracking.

[0114] For example, the dynamic vision sensor 1503 may include a plurality of light sensors and an event generator coupled to the light sensors for sensing dynamic changes in brightness in a scene. The plurality of light sensors are arranged in a matrix of rows and columns, and each light sensor is associated with a row value and a column value. Taking one of the light sensors as an example, the light sensor includes a photodiode connected in series with a resistor between a source voltage and a ground voltage. The voltage across the photodiode is proportional to the intensity (i.e., brightness) of the light incident on the light sensor.

[0115] The light sensor includes a first capacitor connected in parallel with a photodiode. Therefore, the voltage across the first capacitor is the same as the voltage across the photodiode and is proportional to the intensity of the light detected by the light sensor. The light sensor also includes a switch coupled between the first and second capacitors. The second capacitor is coupled between the switch and ground. Therefore, when the switch is closed, the voltage across the second capacitor is the same as the voltage across the first capacitor and is proportional to the intensity of the light detected by the light sensor. When the switch is open, the voltage across the second capacitor is fixed at the voltage level when the switch was last closed.

[0116] The voltages on the first and second capacitors are fed to a comparator. When the difference between the voltages on the first and second capacitors is less than a threshold value, the comparator outputs a unchanged voltage. When the voltage on the first capacitor is at least as high as the threshold value, the comparator outputs a rising voltage. When the voltage on the first capacitor is at least as low as the threshold value, the comparator outputs a falling voltage. When the comparator outputs a unchanged voltage, the event generator performs no operation, indicating that the brightness of the pixels in the real-world scene of the light sensor has not changed. When the comparator outputs a rising or falling voltage, the event generator receives the signal output by the comparator and combines it with the current time and the row and column values ​​associated with the light sensor to generate a corresponding event, outputting an event stream, which can also be called a spatiotemporal data stream.

[0117] Memory 1502 can be used to store computer executable program code, including instructions. Memory 1502 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the monitoring device 1500, etc. Memory 1502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0118] The processor 1501 executes various functional applications and data processing of the monitoring device 1500 by running instructions stored in the memory 1502 and / or instructions stored in the memory configured within the processor. The memory 1502 may also store corresponding program code for target tracking. When the corresponding program code stored in the memory 1502 is executed by the processor 1501, functions such as... Figure 2 The steps of the target tracking method shown are used to determine the position of the target object in the event frame through target tracking.

[0119] Of course, the relevant program code provided in this application embodiment can also be stored in external memory, such as a Micro SD card. In this case, the processor 1501 can run the corresponding program code stored in the external memory through the external memory interface to achieve, for example, Figure 2 The steps of the target tracking method are shown. The external memory interface can be used to connect to external memory, thereby expanding the storage capacity of the monitoring device 1500.

[0120] Transceiver 1504 can implement mobile communication and wireless communication functions. For example, transceiver 1504 may include a mobile communication component for implementing mobile communication functions. The mobile communication component may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication component can receive electromagnetic waves via an antenna, filter and amplify the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication component can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna.

[0121] Transceiver 1504 may include a wireless communication component for implementing wireless communication functionality. The wireless communication component may include a Bluetooth chip for transmitting data with various slave devices. For example, the Bluetooth chip may receive event data or signaling signals with marked target boxes from processor 1501, frequency modulate and amplify them, convert them into electromagnetic waves via an antenna, and transmit them to a server or a terminal device with a display screen. The display screen may show the processing results of the event data to the user.

[0122] Based on the above embodiments, this application also provides a computer program product, including a computer program or instructions, which, when run on a computer, cause the computer to execute any of the target tracking methods provided in the above embodiments.

[0123] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform any of the target tracking methods provided in the above embodiments.

[0124] The storage medium can be any available medium that a computer can access. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0125] Based on the above embodiments, this application also provides a chip for reading a computer program stored in a memory to implement any of the target tracking methods provided in the above embodiments.

[0126] In summary, this application provides a target tracking method and apparatus. In this scheme, the target tracking apparatus obtains template frame event data and search frame event data from event data, and respectively obtains template feature information from the template frame event data and search feature information from the search frame event data. The template frame event data is at least the previous frame of data in the search frame event data, and contains location region information of the target object. After obtaining the template feature information from the template frame event data and the search feature information from the search frame event data, a codec network module combines the template feature information and the search feature information to determine target feature information. This target feature information is then input into a regression network module to obtain the location region information of the target object in the search frame event data. Compared with traditional methods for processing color images, the above process involves less computation and can improve the processing speed of target tracking.

[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of protection of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target tracking method characterized by, The method includes: The template feature information of the template frame event data and the search feature information of the search frame event data are obtained respectively; the template frame event data is at least the previous frame data of the search frame event data, and the template frame event data contains the location region information of the target object; The target feature information is determined by combining the template feature information and the search feature information through the encoding / decoding network module; The target feature information is input into the regression network module to obtain the location region information of the target object in the search frame event data; The encoding / decoding network module includes a motion sensing module, a target sensing module, and a first feature fusion module. The step of determining target feature information by combining the template feature information and the search feature information through the encoding / decoding network module includes: extracting motion feature information of a moving object from the search feature information using the motion sensing module; performing correlation operations on the template feature information and the search feature information to obtain response feature information, and extracting target response feature information of the target object from the response feature information using the target sensing module; and fusing the motion feature information and the target response feature information using the first feature fusion module to obtain the target feature information; or... The encoding / decoding network module includes an event association module, an object motion description module, and a second feature fusion module. The step of determining target feature information by combining the template feature information and the search feature information through the encoding / decoding network module includes: performing correlation operations on the template feature information and the search feature information to obtain target response information; associating local and global features contained in the target response information through the event association module to obtain target response features; extracting object motion features from the target response features through the object motion description module; and fusing the target response features and object motion features through the second feature fusion module to obtain the target feature information.

2. The method of claim 1, wherein, The encoding / decoding network module employs a deep self-attention transform network.

3. The method according to claim 1 or 2, characterized in that, The step of fusing the motion feature information and the target response feature information through the first feature fusion module to obtain the target feature information includes: The first feature fusion module determines similarity information between the motion feature information and the target response feature information, and then performs weighted fusion of the similarity feature information and the motion feature information to obtain the target feature information.

4. The method according to claim 1 or 2, characterized in that, Both the template frame event data and the search frame event data are event data collected by a dynamic visual sensor.

5. The method according to claim 1 or 2, characterized in that, The steps of obtaining template feature information from template frame event data and search feature information from search frame event data respectively include: The template feature information is obtained by extracting features from the template frame event data using the first feature extraction module. The search feature information is obtained by extracting features from the search frame event data using the second feature extraction module.

6. The method according to claim 5, characterized in that, The first feature extraction module and the second feature extraction module share network parameters.

7. A target tracking device, characterized in that, include: Memory and processor; The memory is used to store computer instructions; The processor is configured to read computer instructions from the memory and perform the following steps: The template feature information of the template frame event data and the search feature information of the search frame event data are obtained respectively; the template frame event data is at least the previous frame data of the search frame event data, and the template frame event data contains the location region information of the target object; The target feature information is determined by combining the template feature information and the search feature information through the encoding / decoding network module; The target feature information is input into the regression network module to obtain the location region information of the target object in the search frame event data; The encoding / decoding network module includes a motion perception module, a target perception module, and a first feature fusion module. The processor is specifically configured to: extract motion feature information of a moving object from the search feature information using the motion perception module; perform correlation operations on the template feature information and the search feature information to obtain response feature information, and extract target response feature information of the target object from the response feature information using the target perception module; perform feature fusion on the motion feature information and the target response feature information using the first feature fusion module to obtain the target feature information; or... The encoding / decoding network module includes an event association module, an object motion description module, and a second feature fusion module. The processor is specifically used to: perform correlation operations on the template feature information and the search feature information to obtain target response information; associate the local and global features contained in the target response information through the event association module to obtain target response features; extract object motion features from the target response features through the object motion description module; and perform feature fusion on the target response features and the object motion features through the second feature fusion module to obtain the target feature information.

8. The apparatus according to claim 7, characterized in that, The encoding / decoding network module employs a deep self-attention transform network.

9. The apparatus according to claim 7 or 8, characterized in that, The processor is specifically used for: The first feature fusion module determines similarity information between the motion feature information and the target response feature information, and then performs weighted fusion of the similarity feature information and the motion feature information to obtain the target feature information.

10. The apparatus according to claim 7 or 8, characterized in that, The processor is specifically used for: The template feature information is obtained by extracting features from the template frame event data using the first feature extraction module. The search feature information is obtained by extracting features from the search frame event data using the second feature extraction module.

11. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 6.

12. A computer program product, characterized in that, It includes computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 6.

Images