Target tracking method, target tracking device, computer readable medium

Abstract

The present disclosure provides a target tracking method, comprising: acquiring search neuromorphic data of a target scene, the search neuromorphic data representing light intensity change information of the target scene; inputting the search neuromorphic data into a target tracking model, and acquiring feature data of the target scene by using a pulse neural network in the target tracking model; respectively determining similarity relationships between the feature data of the target scene and feature data of at least one target to be tracked; and determining tracking results of each of the at least one target to be tracked in the target scene according to the similarity relationships between the feature data of the target scene and the feature data of each of the at least one target to be tracked. The present disclosure also provides a target tracking device and a computer readable medium.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and in particular, to a target tracking method, a target tracking device, and a computer readable medium. BACKGROUND

[0002] Target tracking is an important field of computer vision research, and target tracking technology based on deep learning algorithms has made great progress.

[0003] However, the performance of target tracking in some related technologies still needs to be improved. SUMMARY

[0004] The present disclosure provides a target tracking method, a target tracking device, and a computer readable medium.

[0005] In a first aspect, the present disclosure provides a target tracking method, comprising:

[0006] obtaining search neuromorphic data of a target scene, the search neuromorphic data representing light intensity change information of the target scene;

[0007] inputting the search neuromorphic data into a target tracking model, and obtaining feature data of the target scene by using a pulse neural network in the target tracking model;

[0008] respectively determining a similarity relationship between the feature data of the target scene and feature data of at least one target to be tracked;

[0009] determining a tracking result for each of the targets to be tracked in the target scene according to the similarity relationship between the feature data of the target scene and the feature data of each of the targets to be tracked.

[0010] In some embodiments, obtaining search neuromorphic data of a target scene comprises:

[0011] obtaining event data of the target scene based on a neuromorphic vision sensor, the event data of the target scene representing light intensity change information of the target scene;

[0012] grouping the event data of the target scene according to a preset time step to obtain the search neuromorphic data.

[0013] In some embodiments, grouping the event data of the target scene according to a preset time step to obtain the search neuromorphic data comprises:

[0014] determining a starting time point;

[0015] Frame the event data of the target scene in a time range starting from the starting time point and having a length satisfying a first preset condition to obtain the search neuromorphic data.

[0016] In some embodiments, the starting time point is determined, including:

[0017] According to a preset time interval, at least one starting time point is determined.

[0018] In some embodiments, the preset time interval is a first time interval; according to the preset time interval, at least one starting time point is determined, including:

[0019] A starting time point is determined every interval of the first time interval; wherein the first time interval is equal to the time step.

[0020] In some embodiments, the preset time interval is a second time interval; according to the preset time interval, at least one starting time point is determined, including:

[0021] A starting time point is determined every interval of the second time interval; wherein the second time interval is less than the time step.

[0022] In some embodiments, the preset time interval is a third time interval; according to the preset time interval, at least one starting time point is determined, including:

[0023] A candidate time point is determined every interval of the third time interval; wherein the third time interval and the time step satisfy a first preset relationship.

[0024] For any one of the candidate time points, the event data of the target scene in a target time interval is framed to obtain target frame data; wherein the target time interval starts from the candidate time point and has a length satisfying a second preset relationship with the time step.

[0025] According to the target frame data, a probability of the presence of the target to be tracked in the target time interval is determined.

[0026] In a case where the probability of the presence of the target to be tracked in the target time interval satisfies a second preset condition, a starting time point is determined every interval of a fourth time interval in the target time interval; wherein the fourth time interval is less than the time step.

[0027] In some embodiments, the time range having a length satisfying the first preset condition includes a time range having a length equal to the time step.

[0028] In some embodiments, the search neuromorphic data is input into a target tracking model, feature data of the target scene is extracted by using a pulse neural network in the target tracking model, including:

[0029] The spatio-temporal feature data of the search neuromorphic data is extracted by using the pulse neural network;

[0030] The time dimension information in the spatio-temporal feature data is integrated to obtain the feature data of the target scene.

[0031] In some embodiments, the time dimension information in the spatio-temporal feature data is integrated to obtain the feature data of the target scene, including:

[0032] The time dimension information in the spatio-temporal feature data is integrated to obtain the feature data of the target scene.

[0033] In some embodiments, the similarity relationship between the feature data of the target scene and the feature data of at least one to-be-tracked target is determined respectively, including:

[0034] For any one of the to-be-tracked targets, cross-correlation operation is performed on the feature data of the to-be-tracked target and the feature data of the target scene to obtain the similarity relationship between the feature data of the target scene and the feature data of the to-be-tracked target.

[0035] In some embodiments, the cross-correlation operation is performed on the feature data of the to-be-tracked target and the feature data of the target scene to obtain the similarity relationship between the feature data of the target scene and the feature data of the to-be-tracked target, including:

[0036] The feature data of the target scene is convolved with the feature data of the to-be-tracked target as a convolution kernel to obtain a response score matrix, and the response score matrix represents the similarity relationship between the feature data of the target scene and the feature data of the to-be-tracked target.

[0037] In some embodiments, according to the similarity relationship between the feature data of the target scene and the feature data of each to-be-tracked target, a tracking result of each to-be-tracked target in the target scene is determined, including:

[0038] The sub-region in the target scene corresponding to a response score greater than or equal to a similarity threshold in the response score matrix of the to-be-tracked target is determined as the position of the to-be-tracked target in the target scene.

[0039] In some embodiments, the target tracking model includes a twin network, and the twin network includes two input channels, and each input channel corresponds to one pulse neural network.

[0040] Before determining the similarity relationship between the feature data of the target scene and the feature data of the at least one target to be tracked respectively, the method further comprises:

[0041] obtaining template neuro-morphic data of the target to be tracked, the template neuro-morphic data representing light intensity variation information of the target to be tracked;

[0042] inputting the template neuro-morphic data into one input channel of the Siamese network, and obtaining the feature data of the target to be tracked by using the spiking neural network;

[0043] The step of inputting the search neuro-morphic data into the target tracking model and obtaining the feature data of the target scene by using the spiking neural network in the target tracking model comprises:

[0044] inputting the search neuro-morphic data into another input channel of the Siamese network, and obtaining the feature data of the target scene by using the spiking neural network.

[0045] In some embodiments, the method further comprises:

[0046] training the Siamese network by using a logical error function.

[0047] In some embodiments, the spiking neural network comprises one or more of a two-dimensional convolution layer, a batch normalization layer, a rectified linear unit activation function layer, a max-pooling layer, a leaky integrate-and-fire layer.

[0048] In a second aspect, the embodiments of the present disclosure provide a target tracking device, comprising:

[0049] a data acquisition module configured to acquire search neuro-morphic data of a target scene, the search neuro-morphic data representing light intensity variation information of the target scene;

[0050] a feature extraction module configured to input the search neuro-morphic data into a target tracking model, and obtain feature data of the target scene by using a spiking neural network in the target tracking model;

[0051] a data processing module configured to determine a similarity relationship between the feature data of the target scene and the feature data of at least one target to be tracked respectively;

[0052] a target tracking module configured to determine a tracking result of each target to be tracked in the target scene according to the similarity relationship between the feature data of the target scene and the feature data of each target to be tracked.

[0053] In a third aspect, the embodiments of the present disclosure provide a target tracking device, comprising:

[0054] one or more processors;

[0055] a memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the target tracking method according to any one of the first aspect of the embodiments of the present disclosure.

[0056] In a fourth aspect, the embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, when the program is executed by a processor, the target tracking method according to any one of the first aspect of the embodiments of the present disclosure is implemented.

[0057] In the embodiments of the present disclosure, when target tracking is performed, search neuromorphic data representing light intensity change information of a target scene is obtained, a target tracking model based on a spiking neural network can obtain feature data of the target scene according to the search neuromorphic data, and then determine a tracking result for each target to be tracked in the target scene according to a similarity relationship between the feature data of the target scene and the feature data of the at least one target to be tracked; wherein the obtained search neuromorphic data representing the light intensity change information of the target scene has less redundancy and low latency; using a spiking neural network to process the search neuromorphic data can save a large number of product operations. Based on the above features, the storage resources required for target tracking can be saved, the operation rate of target tracking is improved, and the power consumption of target tracking is reduced, thereby enhancing the performance of target tracking.

[0058] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the embodiments of the present disclosure are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of the specific example embodiments, with reference to the accompanying drawings, in which:

[0060] Figure 1 is a flowchart of a target tracking method in the embodiments of the present disclosure;

[0061] Figure 2 is a flowchart of part of the steps in another target tracking method in the embodiments of the present disclosure;

[0062] Figure 3is a schematic diagram of determining a starting time point in an embodiment of the present disclosure;

[0063] Figure 4 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0064] Figure 5 is an architecture schematic diagram of acquiring feature data in an embodiment of the present disclosure;

[0065] Figure 6 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0066] Figure 7 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0067] Figure 8 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0068] Figure 9 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0069] Figure 10 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0070] Figure 11 is an architecture schematic diagram of a twin network in an embodiment of the present disclosure;

[0071] Figure 12 is a flowchart of part of steps in another target tracking method in an embodiment of the present disclosure;

[0072] Figure 13 is a component block diagram of a target tracking device in an embodiment of the present disclosure;

[0073] Figure 14 is a component block diagram of a target tracking device in an embodiment of the present disclosure;

[0074] Figure 15 is a component block diagram of a computer readable medium in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] For those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered only as exemplary. Therefore, those skilled in the art should realize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures.

[0076] In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0077] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0078] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "comprise" and / or "consist of", when used in the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "connected" or "coupled" and / or similar terms are not limited to a physical or mechanical connection, but can include an electrical connection, whether direct or indirect.

[0079] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0080] The inventors of the present disclosure have found that some related technologies based on traditional cameras and convolutional neural networks (CNN) for target tracking have the following disadvantages:

[0081] Data acquisition: Traditional cameras sample globally at a fixed frequency, containing a large amount of redundant information, causing resource waste; the resolution of traditional cameras is low, and high-speed moving targets cannot be captured.

[0082] Tracking algorithm: There are a large number of product operations in convolutional neural networks, high power consumption, and slow operation speed; convolutional neural networks cannot utilize the spatiotemporal information of data.

[0083] Therefore, in a first aspect, the present disclosure provides a target tracking method, comprising: Figure 1 The present disclosure provides a target tracking method, comprising:

[0084] S1, acquiring search neuromorphic data of a target scene, the search neuromorphic data representing light intensity change information of the target scene;

[0085] S2, inputting the search neuromorphic data into a target tracking model, and acquiring feature data of the target scene by using a spiking neural network in the target tracking model;

[0086] S3, respectively determining a similarity relationship between the feature data of the target scene and feature data of at least one to-be-tracked target;

[0087] S4, determining a tracking result of each to-be-tracked target in the target scene according to the similarity relationship between the feature data of the target scene and the feature data of each to-be-tracked target.

[0088] In the present disclosure, the search neuromorphic data is neuromorphic data generated by a neuromorphic vision sensor. The neuromorphic vision sensor is a sensor that simulates the working mechanism of biological vision, and can detect light intensity changes in a target scene and output event information representing the light intensity changes. In some embodiments, the neuromorphic vision sensor represents scene light intensity changes in the form of asynchronous spatio-temporal pulse signals. In some embodiments, the neuromorphic vision sensor includes a dynamic vision sensor (DVS).

[0089] It should be noted that, compared with a traditional camera, the neuromorphic vision sensor has the advantages of high temporal resolution, high dynamic range, low power consumption, less data redundancy, and low latency. Based on this, the present disclosure uses the neuromorphic vision sensor to acquire search neuromorphic data, and then inputs the search neuromorphic data into a target tracking model to acquire feature data of the target scene, which can save storage resources, improve operation speed, and reduce power consumption.

[0090] In the present disclosure, a spiking neural network (SNN) is used as the backbone network of the target tracking model. In the spiking neural network, neurons are only activated when the membrane potential reaches a specific threshold, and the neurons in each layer of the spiking neural network transmit pulse sequences with values of 0 or 1. Using the spiking neural network to process the search neuromorphic data can save a large number of multiplication operations, which is conducive to improving the operation speed and reducing the power consumption. In addition, the spiking neural network also has the ability to extract time information, which can enhance the performance of the target tracking model.

[0091] In the embodiments of the present disclosure, tracking can be performed on a single target in the target scene, or tracking can be performed on multiple targets in the target scene. The embodiments of the present disclosure do not make special limitations on this.

[0092] In the embodiments of the present disclosure, the feature data of the target to be tracked and the feature data of the target scene can be in the same form. In some embodiments, the feature data of the target to be tracked and the feature data of the target scene are in the same form, which means that the same processing procedure is performed when obtaining the feature data of the target scene and obtaining the feature data of the target to be tracked. For example, the same processing as in step S1 is performed on the target to be tracked to obtain the template neuromorphic data, and the same processing as in step S2 is performed on the template neuromorphic data to obtain the feature data of the target to be tracked.

[0093] In the embodiments of the present disclosure, the feature data of the target to be tracked can be pre-acquired and stored, or can be acquired in real time. The embodiments of the present disclosure do not make special limitations on this.

[0094] The embodiments of the present disclosure do not make special limitations on the tracking result of the target to be tracked in the target scene. For example, the tracking result can be the position of the target to be tracked in the target scene, or the moving track of the target to be tracked in the target scene, or the position or moving track of the target to be tracked determined on the basis of identification of the target to be tracked.

[0095] In the target tracking method provided by the embodiments of the present disclosure, when the target tracking is performed, the search neuromorphic data representing the light intensity change information of the target scene is acquired, the target tracking model based on the spiking neural network can acquire the feature data of the target scene according to the search neuromorphic data, and then determine the tracking result of each target to be tracked in the target scene according to the similarity relationship between the feature data of the target scene and the feature data of at least one target to be tracked; wherein the search neuromorphic data representing the light intensity change information of the target scene has less redundancy and lower time delay; the search neuromorphic data is processed by using the spiking neural network, which can save a large number of product operations. Based on the above features, the storage resources required for target tracking can be saved, the operation rate of target tracking is improved, and the power consumption of target tracking is reduced, thereby enhancing the performance of target tracking.

[0096] In some embodiments, referring to Figure 2 , step S1 includes:

[0097] S11, acquiring event data of a target scene based on a neuromorphic vision sensor, the event data of the target scene representing light intensity change information of the target scene;

[0098] S12, grouping the event data of the target scene according to a preset time step to obtain search neuromorphic data.

[0099] In some embodiments, the neuromorphic visual sensor outputs event data in the form of (x, y, t, p) quadruples, each of which corresponds to an event. Among them, the coordinate component (x, y) represents the coordinates of the event; the time component t represents the occurrence time of the event; and the polarity component p represents the polarity of the event, that is, whether the illumination is enhanced or weakened, for example, 0 represents that the illumination is weakened, and 1 represents that the illumination is enhanced.

[0100] In some embodiments, the step S12 of grouping the event data of the target scene according to the preset time step can be: as an example, the preset time step is represented by T, and the event data in the form of (T, C, H, W) is encapsulated into frame data with the dimension of (T, C, H, W) according to the time step T, wherein C represents Channel, H represents Height of the frame data, and W represents Width of the frame data.

[0101] In this way, the search neuromorphic data obtained through steps S11 to S12 carries time information, so that the search neuromorphic data can be subjected to spatio-temporal feature extraction by the spiking neural network in step S2, which can further enhance the target tracking performance compared with the convolutional neural network which cannot utilize the spatio-temporal information of the data.

[0102] The embodiments of the present disclosure do not make special limitations on how to group the event data of the target scene according to the preset time step.

[0103] In some embodiments, grouping the event data of the target scene according to the preset time step to obtain the search neuromorphic data comprises:

[0104] determining a starting time point;

[0105] grouping the event data of the target scene in a time range with the starting time point as the starting point and a length satisfying a first preset condition to obtain the search neuromorphic data.

[0106] It should be noted that, in the embodiments of the present disclosure, by reasonably selecting the starting time point for grouping, the success rate of target tracking can be improved, thereby enhancing the performance of target tracking.

[0107] The embodiments of the present disclosure do not make special limitations on the first preset condition.

[0108] In some embodiments, the time range with the length satisfying the first preset condition comprises: a time range with a length equal to the time step.

[0109] The embodiments of the present disclosure do not make special limitations on how to determine the starting time point.

[0110] In some embodiments, the starting time point is determined according to a preset time interval.

[0111] The at least one starting time point is determined according to a preset time interval.

[0112] In the embodiments of the present disclosure, the greater the preset time interval is, the smaller the data amount of the search neuromorphic data obtained by grouping the event data of the target scene is, and the smaller the data amount is, the smaller the operation amount of the target tracking is and the lower the accuracy is; the smaller the preset time interval is, the greater the data amount of the search neuromorphic data obtained by grouping the event data of the target scene is, and the greater the data amount is, the greater the operation amount of the target tracking is and the higher the accuracy is.

[0113] In the embodiments of the present disclosure, by reasonably selecting the preset time interval, the operation amount and the accuracy of the target tracking can be balanced, and the target tracking performance can be improved.

[0114] In some embodiments, the starting time points are sequentially selected at intervals of the time step T and grouped to obtain frame data that does not overlap in the time dimension.

[0115] Correspondingly, in some embodiments, the preset time interval is a first time interval; the at least one starting time point is determined according to the preset time interval, including:

[0116] A starting time point is determined every interval of the first time interval; and the first time interval is equal to the time step.

[0117] It should be noted that, in the embodiments of the present disclosure, the starting time points are sequentially selected at intervals of the time step T and grouped to avoid frame data redundancy, which is conducive to reducing the calculation amount and saving the calculation resources.

[0118] In some embodiments, a plurality of starting time points are sequentially selected and grouped in a time interval with a length equal to the time step T to obtain frame data that overlaps in the time dimension.

[0119] Correspondingly, in some embodiments, the preset time interval is a second time interval; the step of determining the at least one starting time point according to the preset time interval includes:

[0120] A starting time point is determined every interval of the second time interval; and the second time interval is less than the time step.

[0121] The length of the second time interval is not specially limited in the embodiments of the present disclosure. For example, the length of the second time interval can be equal to T / 8, T / 4, T / 2, etc.

[0122] It should be noted that, in the embodiments of the present disclosure, the plurality of starting time points are sequentially selected in the time intervals with the length equal to the time step T, and the frame grouping is performed respectively, which can improve the accuracy of target tracking.

[0123] In some embodiments, the preset time interval is a third time interval; and the at least one starting time point is determined according to the preset time interval, including:

[0124] A candidate time point is determined every third time interval; wherein the third time interval and the time step satisfy a first preset relationship.

[0125] As an example, the first preset relationship can be that the third time interval is equal to the time step.

[0126] For any candidate time point, the event data of the target scene in a target time interval is framed to obtain target frame data; wherein the target time interval takes the candidate time point as a starting point, and the length and the time step satisfy a second preset relationship.

[0127] As an example, the second preset relationship can be that the target time interval takes the candidate time point as a starting point, and the length is equal to the time step.

[0128] The probability that a target to be tracked exists in the target time interval is determined according to the target frame data.

[0129] In the case that the probability that the target to be tracked exists in the target time interval satisfies a second preset condition, a starting time point is determined in the target time interval every fourth time interval; wherein the fourth time interval is less than the time step.

[0130] As an example, the second preset condition can include that the probability that the target to be tracked exists in the target time interval is greater than a probability threshold, and the specific value of the probability threshold is not limited in the embodiments of the present disclosure.

[0131] In combination with Figure 3 Further explanation and description. As Figure 3As shown, t1, t2, t3, and t4 are candidate time points. The length of time interval 0 between t1 and t2 is equal to the time step T, the length of time interval 1 between t2 and t3 is equal to the time step T, and the length of time interval 2 between t3 and t4 is equal to the time step T. For candidate time point t1, the event data of the target scene within time interval 0 is framed. If the probability of the target to be tracked existing within time interval 0 is not greater than the probability threshold, the next candidate time point t2 is selected. It should be noted that for candidate time point t1, coarse matching is performed. During coarse matching, frame inference is performed every time step T. It should be noted that coarse matching is a concept relative to fine matching. The matching accuracy of coarse matching is lower than that of fine matching, but the data redundancy of coarse matching is also lower than that of fine matching.

[0132] For candidate time point t2, the event data of the target scene within time interval 1 are framed. If the probability of the target to be tracked existing within time interval 1 is greater than the probability threshold, then multiple starting time points are redefined within time interval 1. For example, Figure 3 Within time interval 1, eight starting time points were determined, and frames were generated to obtain eight sets of frame data: frame data 0, frame data 1, ..., frame data 7. It should be noted that for candidate time point t2, a fine-matching process was performed, enabling precise tracking along the time dimension.

[0133] For candidate time point t3, during the process of fine matching for candidate time point t2, it is equivalent to simultaneously performing fine matching in time interval 2, so there is no need to perform coarse matching.

[0134] like Figure 3 The proposed scheme for determining the starting time point can reduce computational load and save computational resources when there is no target to track, and can maintain tracking accuracy in the time dimension when there is a target to track, thereby improving target tracking performance.

[0135] Accordingly, in some embodiments, reference is made to Figure 4 Step S2 includes:

[0136] S21. Use spiking neural networks to extract spatiotemporal feature data of search neuromorphic data;

[0137] S22. Integrate the time dimension information in the spatiotemporal feature data to obtain the feature data of the target scene.

[0138] Figure 5 This is a schematic diagram of the architecture for acquiring feature data of a target scene in some embodiments. For example... Figure 5 As shown, the event data (x,y,t,p)1……(x,y,t,p) are divided according to the time step T. i… (x, y, t, p) n packaging search neuromorphic data into a dimension of (T, C, H, W); through a spiking neural network spatiotemporal feature extraction is performed on the search neuromorphic data to obtain spatiotemporal feature data; then, time dimension information in the spatiotemporal feature data is integrated through a Sum Layer to obtain feature data of the target scene.

[0139] In some embodiments, the spatiotemporal feature data is dimension-reduced by averaging the spatiotemporal feature data in the time dimension through the Sum Layer, so that the dimension of the feature data of the target scene is (C, H, W).

[0140] Correspondingly, in some embodiments, with reference to Figure 6 , step S22 includes:

[0141] S221, data averaging processing is performed on the spatiotemporal feature data in the time dimension to obtain the feature data of the target scene.

[0142] In the disclosed embodiments, the data averaging processing is performed on the spatiotemporal feature data in the time dimension, so that the spatiotemporal feature data is dimension-reduced, the information of the time dimension can be integrated, and the target tracking performance can be enhanced by using spatiotemporal information.

[0143] In some embodiments, the data averaging processing on the spatiotemporal feature data in the time dimension can include: performing average pooling with a predetermined time step in the time dimension. It should be understood that other ways can also be used by those skilled in the art to realize dimension reduction of the spatiotemporal feature data, and the present disclosure does not limit this.

[0144] In some embodiments, with reference to Figure 7 , step S3 includes:

[0145] S31, for any one target to be tracked, cross-correlation operation is performed on the feature data of the target to be tracked and the feature data of the target scene to obtain a similarity relationship between the feature data of the target scene and the feature data of the target to be tracked.

[0146] In the embodiments of the present disclosure, the similarity relationship between the feature data of the target scene and the feature data of the target to be tracked is determined based on the cross-correlation operation, which is beneficial to improving the accuracy of similarity judgment and further enhancing the target tracking performance.

[0147] The embodiments of the present disclosure do not specially limit the cross-correlation operation in step S31.

[0148] In some embodiments, with reference to Figure 8 , step S31 includes:

[0149] S311, convolve the feature data of the target scene with the feature data of the target to be tracked as a convolution kernel to obtain a response score matrix, the response score matrix representing the similarity relationship between the feature data of the target scene and the feature data of the target to be tracked.

[0150] It should be noted that each element in the response score matrix can be a response score, and the response score can represent the similarity between the part in the feature data of the corresponding target scene and the feature data of the target to be tracked, and thus can represent the similarity between the sub-region in the corresponding target scene and the target to be tracked.

[0151] In some embodiments, the response score is positively correlated with the similarity between the sub-region in the corresponding target scene and the target to be tracked, and the greater the response score, the higher the similarity between the sub-region in the corresponding target scene and the target to be tracked.

[0152] Correspondingly, in some embodiments, with reference to Figure 9 , step S4 comprises:

[0153] S41, determining the sub-region in the target scene corresponding to the response score greater than or equal to the similarity threshold in the response score matrix of the target to be tracked as the position of the target to be tracked in the target scene.

[0154] In the embodiments of the present disclosure, by reasonably determining the similarity threshold, the similarity judgment can meet the demand, thereby enhancing the target tracking performance.

[0155] In some embodiments, the sub-region in the target scene corresponding to all the response scores greater than or equal to the similarity threshold in the response score matrix is determined as the position of the target to be tracked in the target scene.

[0156] In some embodiments, the sub-region in the target scene corresponding to the largest one of the response scores greater than or equal to the similarity threshold in the response score matrix is determined as the position of the target to be tracked in the target scene.

[0157] In some embodiments, the target tracking model comprises a Siamese Network, and the Siamese Network comprises two input channels, each input channel corresponding to a spiking neural network.

[0158] With reference to Figure 10 , before step S3, the above method further comprises:

[0159] S51, obtaining template neuromorphic data of the target to be tracked, the template neuromorphic data representing light intensity change information of the target to be tracked;

[0160] S52, input the template neuro morphology data into one input channel of the Siamese network, and acquire feature data of the target to be tracked by using the spiking neural network;

[0161] With reference to Figure 10 , step S2 comprises:

[0162] S23, input the search neuro morphology data into another input channel of the Siamese network, and acquire feature data of the target scene by using the spiking neural network.

[0163] In the embodiments of the present disclosure, based on the target tracking model of the Siamese network, the feature data of the target to be tracked and the feature data of the target scene can be synchronously acquired through two input channels of the Siamese network respectively, the accuracy of acquiring the feature data of the target to be tracked and the feature data of the target scene is ensured, and thus the efficiency of target tracking is improved and the performance of target tracking is enhanced.

[0164] In some embodiments, the architecture of the Siamese network is as shown in Figure 11 .

[0165] In Figure 11 , the event data (x, y, t, p) is packaged into template neuro morphology data with a dimension of (T, C1, H1, W1) according to the time step T. 11 … (x, y, t, p) 1i … (x, y, t, p) 1n The template neuro morphology data is packaged into search neuro morphology data with a dimension of (T, C2, H2, W2) according to the time step T. The search neuro morphology data is subjected to spatio-temporal feature extraction by the spiking neural network to obtain spatio-temporal feature data of the search neuro morphology data; and then the time dimension information in the spatio-temporal feature data of the search neuro morphology data is integrated by the Sum Layer to reduce the dimension, and the feature data of the target scene with a dimension of (C2, H2, W2) is obtained.

[0166] In Figure 11 , the event data (x, y, t, p) is packaged into template neuro morphology data with a dimension of (T, C1, H1, W1) according to the time step T. 21 … (x, y, t, p) 2i … (x, y, t, p) 2n The search neuro morphology data is packaged into search neuro morphology data with a dimension of (T, C2, H2, W2) according to the time step T. The search neuro morphology data is subjected to spatio-temporal feature extraction by the spiking neural network to obtain spatio-temporal feature data of the search neuro morphology data; and then the time dimension information in the spatio-temporal feature data of the search neuro morphology data is integrated by the Sum Layer to reduce the dimension, and the feature data of the target scene with a dimension of (C2, H2, W2) is obtained.

[0167] As Figure 11As shown, the feature data of the target to be tracked and the feature data of the target scene are cross-correlated to obtain a response score matrix representing the similarity relationship between the feature data of the target to be tracked and the feature data of the target scene.

[0168] In some embodiments, referring to Figure 12 The method further includes:

[0169] S6, training the twin network using a logistic error function.

[0170] In the embodiments of the present disclosure, the twin network is trained using a logistic error function, which can improve the training accuracy and thus improve the performance of the trained twin network, i.e., enhance the target tracking performance.

[0171] In some embodiments, training the twin network using a logistic error function includes:

[0172] The similarity between the corresponding positions of the response score matrix (Score Map) and the effective value label (Ground Truth Label) is calculated using formula (1):

[0173] l(y,v)=log(1+exp(-yv)) (1)

[0174] Wherein, y corresponds to Ground Truth Label, and v corresponds to Score Map.

[0175] The loss sum of all corresponding points of Score Map and Ground Truth Label is calculated using formula (2), and the average is taken:

[0176]

[0177] The parameters of the twin network are updated according to the calculation result.

[0178] In some embodiments, as shown in Figure 11 The spiking neural network includes one or more of a two-dimensional convolution layer (Conv2D), a batch normalization layer (Batch Normalization), a rectified linear unit activation function layer (ReLU, Rectified Linear Units), a maximum pooling layer (MaxPool), and a leaky integrate and fire layer (LIF, Leaky Integrate and Fire).

[0179] In some embodiments, as shown in Figure 11 The spiking neural network includes Conv2D, Batch Normalization, ReLU, MaxPool, and LIF.

[0180] It should be noted that, in the pulse neural network as shown in Figure 11 The neuron of the pulse neural network is a LIF neuron:

[0181] The synaptic connection between neurons is shown in formula (3):

[0182] I t = Conv(X t ,W) (3)

[0183] Wherein, I t represents a current signal, X t represents a pulse signal, and W represents a weight.

[0184] The neuron calculates the membrane potential according to formula (4):

[0185]

[0186] Wherein, represents the current membrane potential, represents the membrane potential at the last time, and r represents the impedance.

[0187] The neuron determines whether to fire a pulse according to formula (5):

[0188]

[0189] Wherein, F t represents a firing pulse, and V th represents a firing threshold. Formula (5) indicates that when the membrane potential is greater than or equal to the firing threshold, the neuron fires; when the membrane potential is less than the firing threshold, the neuron does not fire.

[0190] After the neuron fires, the membrane potential is reset according to formula (6):

[0191]

[0192] Leakage is performed according to formula (7):

[0193]

[0194] Wherein, a represents a multiplicative decay, and β represents an additive decay.

[0195] The neuron input is represented by formula (8):

[0196] Y t = F t (8)

[0197] In a second aspect, with reference to Figure 13 The present application provides a target tracking device, comprising:

[0198] The data acquisition module 101 is configured to acquire search neuromorphic data of a target scene, the search neuromorphic data representing light intensity change information of the target scene.

[0199] The feature extraction module 102 is configured to input the search neuromorphic data into a target tracking model, and acquire feature data of the target scene by using a pulse neural network in the target tracking model.

[0200] The data processing module 103 is configured to determine a similarity relationship between the feature data of the target scene and feature data of at least one to-be-tracked target.

[0201] The target tracking module 104 is configured to determine a tracking result of each to-be-tracked target in the target scene according to the similarity relationship between the feature data of the target scene and the feature data of each to-be-tracked target.

[0202] In a third aspect, a target tracking device is provided, including: Figure 14 In a third aspect, a target tracking device is provided, including:

[0203] one or more processors 201;

[0204] a memory 202, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the target tracking method of any one of the first aspect of the embodiments of the present disclosure;

[0205] one or more I / O interfaces 203 connected between the processor and the memory, configured to implement information interaction of the processor and the memory.

[0206] The processor 201 is a device with data processing capability, including but not limited to a central processing unit (CPU) and the like; the memory 202 is a device with data storage capability, including but not limited to a random access memory (RAM, more specifically SDRAM, DDR, etc.), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory (FLASH); the I / O interface (read-write interface) 203 is connected between the processor 201 and the memory 202, and can realize information interaction of the processor 201 and the memory 202, including but not limited to a data bus (Bus) and the like.

[0207] In some embodiments, the processor 201, the memory 202 and the I / O interface 203 are connected with each other through the bus 204, and further connected with other components of the computing device.

[0208] In a third aspect, a target tracking device is provided, including: Figure 15The computer readable medium stores a computer program, and the computer program is executed by a processor to implement the target tracking method according to any one of the first aspect of the embodiments of the present disclosure.

[0209] Those of ordinary skill in the art will understand that all or some of the steps in the methods disclosed above, the functions of the modules / units in the systems and devices can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementation, the division between the modules / units referred to in the above description does not necessarily correspond to the division between physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components working together. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it is common knowledge to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and can include any information delivery media.

[0210] Example embodiments have been disclosed herein and, although the use of specific terms is expressly used herein, they are intended in a generic sense only and are not intended to limit the scope of the present disclosure. In some instances, it can be clear to those of ordinary skill in the art that features, characteristics and / or elements described in connection with a particular embodiment can be used in conjunction with other embodiments, unless explicitly stated otherwise. Accordingly, those of ordinary skill in the art will understand that various changes in form and details can be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A target tracking method, comprising: acquiring search neuromorphic data of a target scene, the search neuromorphic data representing light intensity change information of the target scene; inputting the search neuromorphic data into a target tracking model, and acquiring feature data of the target scene by using a spiking neural network in the target tracking model; respectively determining similarity relationships between the feature data of the target scene and feature data of at least one target to be tracked; determining a tracking result for each of the at least one target to be tracked in the target scene according to the similarity relationships between the feature data of the target scene and the feature data of each of the at least one target to be tracked; wherein the acquiring of the search neuromorphic data of the target scene comprises: determining at least one starting time point according to a preset time interval; based on the starting time point, grouping frames of time data of the target scene according to a preset time interval and a preset time step to obtain the search neuromorphic data; the preset time interval comprises a third time interval; and the determining of the at least one starting time point according to the preset time interval comprises: determining a candidate time point every third time interval, wherein the third time interval and the time step satisfy a first preset relationship; for any one of the candidate time points, grouping frames of event data of the target scene in a target time interval to obtain target frame data, wherein the target time interval has the candidate time point as a starting point and a length satisfying a second preset relationship with the time step; determining a probability that the target to be tracked exists in the target time interval according to the target frame data; in a case where the probability that the target to be tracked exists in the target time interval satisfies a second preset condition, determining a starting time point every fourth time interval in the target time interval, wherein the fourth time interval is smaller than the time step, and the fourth time interval is smaller than the third time interval.

2. The object tracking method of claim 1, wherein, The acquiring of the search neuromorphic data of the target scene comprises: acquiring event data of the target scene based on a neuromorphic vision sensor, the event data of the target scene representing light intensity change information of the target scene; grouping frames of the event data of the target scene according to the time step to obtain the search neuromorphic data.

3. The target tracking method of claim 2, wherein, The grouping frames of the event data of the target scene according to the time step to obtain the search neuromorphic data comprises: determining a starting time point; grouping frames of the event data of the target scene in a time range having the starting time point as a starting point and a length satisfying a first preset condition to obtain the search neuromorphic data.

4. The target tracking method according to claim 1 or 3, wherein, The time range having the length satisfying the first preset condition comprises a time range having a length equal to the time step.

5. The object tracking method of claim 2, wherein, The inputting of the search neuromorphic data into the target tracking model and the extracting of the feature data of the target scene by using the spiking neural network in the target tracking model comprises: extracting spatio-temporal feature data of the search neuromorphic data by using the spiking neural network; integrating time dimension information in the spatio-temporal feature data to obtain the feature data of the target scene.

6. The target tracking method of claim 5, wherein, Integrating time dimension information in the spatio-temporal feature data to obtain the feature data of the target scene, comprising: Performing data averaging processing on the spatio-temporal feature data in the time dimension to obtain the feature data of the target scene.

7. The object tracking method according to any one of claims 1 to 3, 5, 6, wherein, Respectively determining the similarity relationship between the feature data of the target scene and the feature data of at least one to-be-tracked target, comprising: For any one of the to-be-tracked targets, performing cross-correlation operation on the feature data of the to-be-tracked target and the feature data of the target scene to obtain the similarity relationship between the feature data of the target scene and the feature data of the to-be-tracked target.

8. The object tracking method of claim 7, wherein, Performing cross-correlation operation on the feature data of the to-be-tracked target and the feature data of the target scene to obtain the similarity relationship between the feature data of the target scene and the feature data of the to-be-tracked target, comprising: Convoluting the feature data of the target scene with the feature data of the to-be-tracked target as a convolution kernel to obtain a response score matrix, the response score matrix representing the similarity relationship between the feature data of the target scene and the feature data of the to-be-tracked target.

9. The target tracking method of claim 8, wherein, According to the similarity relationship between the feature data of the target scene and the feature data of each of the to-be-tracked targets, determining the tracking result of each of the to-be-tracked targets in the target scene, comprising: Determining the sub-region in the target scene corresponding to the response score greater than or equal to a similarity threshold in the response score matrix of the to-be-tracked target as the position of the to-be-tracked target in the target scene.

10. The object tracking method according to any one of claims 1 to 3, 5, 6, wherein, The target tracking model comprises a twin network, and the twin network comprises two input channels, each of which corresponds to one of the spiking neural networks. Before respectively determining the similarity relationship between the feature data of the target scene and the feature data of at least one to-be-tracked target, the method further comprises: Obtaining template neuromorphic data of the to-be-tracked target, the template neuromorphic data representing light intensity change information of the to-be-tracked target; Inputting the template neuromorphic data into one of the input channels of the twin network to obtain the feature data of the to-be-tracked target by using the spiking neural network; Wherein, the step of inputting the search neuromorphic data into the target tracking model and obtaining the feature data of the target scene by using the spiking neural network in the target tracking model comprises: Inputting the search neuromorphic data into the other input channel of the twin network to obtain the feature data of the target scene by using the spiking neural network.

11. The object tracking method of claim 10, wherein, The method further comprises: Training the twin network using a logical error function.

12. The object tracking method according to any one of claims 1 to 3, 5, 6, wherein, The spiking neural network comprises one or more of a two-dimensional convolution layer, a batch normalization layer, a rectified linear unit activation function layer, a maximum pooling layer, a leaky integrate-and-fire layer.

13. A target tracking device, comprising: A data acquisition module configured to acquire search neuromorphic data of a target scene, the search neuromorphic data representing light intensity change information of the target scene; The feature extraction module is configured to input the search neuromorphic data into a target tracking model, and obtain feature data of the target scene by using a spiking neural network in the target tracking model; The data processing module is configured to determine a similarity relationship between the feature data of the target scene and feature data of at least one to-be-tracked target; The target tracking module is configured to determine a tracking result of each to-be-tracked target in the target scene according to the similarity relationship between the feature data of the target scene and the feature data of each to-be-tracked target. The search neuromorphic data of the target scene comprises: determining at least one starting time point according to a preset time interval; based on the starting time point, grouping frames of time data of the target scene according to a preset time interval and a preset time step to obtain the search neuromorphic data; The preset time interval comprises a third time interval; and the determining of the at least one starting time point according to the preset time interval comprises: determining a candidate time point every third time interval; wherein the third time interval and the time step satisfy a first preset relationship; for any one of the candidate time points, grouping frames of event data of the target scene in a target time interval to obtain target frame data; wherein the target time interval has the candidate time point as a starting point, and a length thereof satisfies a second preset relationship with the time step; determining a probability that the to-be-tracked target exists in the target time interval according to the target frame data; in a case where the probability that the to-be-tracked target exists in the target time interval satisfies a second preset condition, determining a starting time point every fourth time interval in the target time interval; wherein the fourth time interval is smaller than the time step, and the fourth time interval is smaller than the third time interval.

14. A target tracking apparatus, comprising: one or more processors; a memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the target tracking method according to any one of claims 1 to 12.

15. A computer readable medium having a computer program stored thereon, when the program is executed by a processor, the program implements the target tracking method according to any one of claims 1 to 12.

Images