Dynamic vision-single photon sensor fused dynamic target tracking identification method

Abstract

The invention provides a dynamic vision-single photon sensor fused dynamic target tracking identification method, relates to the technical field of visual information processing, and aims to solve the technical problems of low processing speed and high energy consumption of a traditional visual system. The method comprises the steps that a first event pulse corresponding to a static background in the dynamic vision sensor is recognized and removed, a second event pulse is obtained, and the first event pulse is generated through relative movement of the static background and the dynamic vision sensor; the second event pulses are input into the lightweight attention prediction pulse neural network in a sequence form for prediction, and an attention thermodynamic diagram is obtained; cutting the grayscale image of the single-photon sensor according to the distribution characteristics of the attention thermodynamic diagram; inputting the cut grayscale image into a convolutional neural network for identification to obtain an identification result; and fusing the attention thermodynamic diagram and the recognition result to obtain a tracking recognition result of the target.

Description

Technical Field

[0001] The present invention relates to the technical field of visual information processing, and more specifically, to a dynamic target tracking and recognition method based on dynamic vision-single-photon sensor fusion. Background Art

[0002] At present, traditional target tracking and recognition systems mainly rely on static complementary metal oxide semiconductor (CMOS) vision cameras. However, these cameras face serious motion blur problems in high-speed dynamic scenes. At the same time, the processing volume of static frame images is large, resulting in slow processing speed and high energy consumption.

[0003] Dynamic Vision Sensors (DVS) and Single Photon Avalanche Diodes (SPADs) perform well in high-speed, dynamic scenes. The former can output dynamic scene information in the form of event pulses, while the latter can output pulse frames of the scene at extremely high frame rates. However, there is currently a lack of efficient methods for processing this type of information. Existing technologies typically restore DVS and SPAD pulse information to regular frame images through pulse accumulation, which is then processed using convolutional neural networks. This approach, however, loses the inherent pulse sparseness and rich temporal information of DVS and SPADs. Consequently, traditional vision systems still suffer from the disadvantages of slow processing speed and high energy consumption. Summary of the Invention

[0004] In view of this, the present invention provides a dynamic target tracking and recognition method based on dynamic vision and single-photon sensor fusion, aiming to solve the technical problems of slow processing speed and high energy consumption in traditional vision systems.

[0005] One aspect of the present invention provides a dynamic target tracking and recognition method using dynamic vision and single-photon sensor fusion, comprising: identifying a first event pulse corresponding to a static background in a dynamic vision sensor, and performing a elimination process to obtain a second event pulse, wherein the first event pulse is generated by the relative motion of the static background and the dynamic vision sensor; inputting the second event pulse in the form of a sequence into a lightweight attention prediction pulse neural network for prediction to obtain an attention heat map; cropping the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map; inputting the cropped grayscale image into a convolutional neural network for recognition to obtain a recognition result; and fusing the attention heat map and the recognition result to obtain a target tracking and recognition result.

[0006] According to an embodiment of the present invention, the first event pulse corresponding to the static background in the dynamic vision sensor is identified and eliminated before obtaining the second event pulse, including: using an inertial measurement unit to obtain the acceleration of the dynamic vision sensor; integrating the acceleration to obtain speed information of the dynamic vision sensor.

[0007] According to an embodiment of the present invention, identifying a first event pulse corresponding to a static background in a dynamic vision sensor and performing a elimination process to obtain a second event pulse includes: compressing the event pulses of the dynamic vision sensor into the same sequence, wherein the pulse frequency of the same sequence is positively correlated with the movement speed of the target; inputting the compressed event pulse sequence into a pulse neuron model for pulse filtering processing, wherein the activation threshold of the pulse neuron model is determined by speed information; based on the activation threshold, eliminating the first event pulse below the activation threshold to obtain a second event pulse above the activation threshold.

[0008] According to an embodiment of the present invention, the second event pulse is input into the lightweight attention prediction pulse neural network in the form of a sequence for prediction, and the attention heat map is obtained, including: position encoding the second event pulse; time encoding the second event pulse after position encoding; using the self-attention mechanism to calculate the key-value pair attention between the second event pulse sequence after time encoding; based on the key-value pair attention, using a multi-layer perceptron to predict the attention heat map at the current moment.

[0009] According to an embodiment of the present invention, position encoding of the second event pulse includes: using a learnable encoding matrix to perform encoding expansion on the input second event pulse sequence of length n, wherein the encoding expansion method includes:

[0010]

[0011] Among them, π t represents the result after the event pulse sequence is expanded, x t represents the event pulse sequence, W p Represents a learnable encoding matrix, R represents a real number, and d represents the expanded dimension.

[0012] According to an embodiment of the present invention, time-coding the position-coded second event pulse includes: time-coding the position-coded second event pulse using a sine function and a cosine function of different frequencies, wherein the time coding method includes:

[0013]

[0014]

[0015] Where TE represents the time code and t represents the timestamp.

[0016] According to an embodiment of the present invention, cropping the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map includes: generating a confidence matrix, wherein the confidence matrix is ​​the same as the pixel size of the single-photon sensor; increasing the confidence of the area near the confidence matrix according to the distribution characteristics of the attention heat map; activating the single-photon sensor and the corresponding convolutional neural network in response to the confidence reaching a confidence threshold; processing the attention heat map through a clustering algorithm to obtain attention distribution coordinates; and cropping the grayscale image of the single-photon sensor according to the attention distribution coordinates.

[0017] Another aspect of the present invention provides a dynamic target tracking and recognition system of dynamic vision-single photon sensor fusion, including: a background pulse removal module, used to identify the first event pulse corresponding to the static background in the dynamic vision sensor, and perform a removal process to obtain a second event pulse, wherein the first event pulse is generated by the relative motion of the static background and the dynamic vision sensor; an attention prediction pulse neural network module, used to input the second event pulse in the form of a sequence into a lightweight attention prediction pulse neural network for prediction to obtain an attention heat map; an attention-based grayscale image downsampling module, used to crop the grayscale image of the single photon sensor according to the distribution characteristics of the attention heat map; a target recognition convolutional neural network module, used to input the cropped grayscale image into the convolutional neural network for recognition to obtain a recognition result; a fusion module, used to fuse the attention heat map and the recognition result to obtain a target tracking and recognition result.

[0018] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0019] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0020] Another aspect of the present invention provides a computer program product, which includes computer executable instructions. When the instructions are executed, they are used to implement the above method.

[0021] Compared with the prior art, the dynamic target tracking and recognition method based on dynamic vision and single-photon sensor fusion provided by the present invention has at least the following beneficial effects:

[0022] (1) The dynamic target tracking and recognition method based on dynamic vision and single-photon sensor fusion provided by the embodiment of the present invention fully utilizes the characteristics of sparse pulses and rich temporal information of dynamic vision sensors through a unique attention prediction pulse neural network, reduces the overall power consumption and parameter quantity of the system in terms of the computational amount of the neural network model, the amount of computational data, and the computational time, and is suitable for deployment in edge scenarios with limited hardware resources.

[0023] (2) The dynamic target tracking and recognition method of dynamic vision-single-photon sensor fusion provided in the embodiment of the present invention concisely realizes the elimination of specific frequency pulses through pulse neurons, that is, it recognizes and distinguishes static objects and dynamic objects in the scene. The event pulses generated by the relative motion of static objects are useless for the intelligent vision task of target tracking and recognition. By eliminating them, the amount of data that needs to be processed later is greatly reduced.

[0024] (3) The dynamic target tracking and recognition method of dynamic vision-single photon sensor fusion provided by the embodiment of the present invention processes the event pulse sequence directly by the attention prediction pulse neural network, compared with the traditional processing method of accumulating the sequence into frames, which maintains the sparsity of information while also retaining the temporal sequence information. The temporal sequence information of the sequence is processed by neurons with memory characteristics, thereby having lower computational complexity while maintaining higher network performance.

[0025] (4) The dynamic target tracking and recognition method based on dynamic vision and single-photon sensor fusion provided by the embodiment of the present invention uses a lightweight small network with smaller parameters and faster speed to control the convolutional neural network with larger parameters and slower speed, so that the convolutional neural network no longer remains in a working state for a long time, thereby reducing the overall power consumption of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a dynamic target tracking and recognition method using dynamic vision and single-photon sensor fusion according to an embodiment of the present invention is schematically shown;

[0028] Figure 2 The structure block diagram of the dynamic target tracking and recognition system using dynamic vision and single-photon sensor fusion according to an embodiment of the present invention is schematically shown;

[0029] Figure 3 Schematically shows a structural block diagram of an attention prediction spiking neural network module according to an embodiment of the present invention;

[0030] Figure 4Schematically shows the input and output effect diagram of the attention prediction pulse neural network module according to an embodiment of the present invention;

[0031] Figure 5 The structural block diagram of an electronic device suitable for implementing a dynamic target tracking and recognition method of dynamic vision-single photon sensor fusion according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0037] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0038] Figure 1 The flowchart of the dynamic target tracking and recognition method based on dynamic vision-single photon sensor fusion according to an embodiment of the present invention is schematically shown.

[0039] like Figure 1 As shown, the dynamic target tracking and recognition method of dynamic vision-single photon sensor fusion in this embodiment may include operations S1 to S5.

[0040] In operation S1 , a first event pulse corresponding to a static background in a dynamic vision sensor is identified and eliminated to obtain a second event pulse, wherein the first event pulse is generated by relative motion between the static background and the dynamic vision sensor.

[0041] In operation S2, the second event pulse is input into the lightweight attention prediction pulse neural network in the form of a sequence for prediction to obtain an attention heat map.

[0042] In operation S3 , the grayscale image of the single-photon sensor is cropped according to the distribution characteristics of the attention heat map.

[0043] In operation S4, the cropped grayscale image is input into a convolutional neural network for recognition to obtain a recognition result.

[0044] In operation S5, the attention heat map and the recognition result are fused to obtain the target tracking and recognition result.

[0045] In this embodiment, the static background of the event data generated by the dynamic vision sensor (DVS) is first identified and eliminated (that is, the first event pulse is identified and eliminated); then the eliminated event data (that is, the second event pulse) is directly input into the lightweight attention prediction pulse neural network (SNN attention network) in the form of a sequence to obtain a real-time attention heat map; then, the single photon sensor (SPAD) and the corresponding processing network are controlled to be turned on according to the distribution of the heat map, and the SPAD grayscale image is cropped; then, the cropped image is input into the convolutional neural network (CNN recognition network) for recognition; finally, the target tracking and recognition result is obtained by combining the attention heat map and the recognition result.

[0046] The dynamic target tracking and recognition method of dynamic vision-single-photon sensor fusion provided by the embodiment of the present invention fully utilizes the characteristics of sparse pulses and rich temporal information of dynamic vision sensors through a unique attention prediction pulse neural network, reduces the overall power consumption and parameter quantity of the system in terms of the computational amount of the neural network model, as well as the amount of computational data and computational time, and is suitable for deployment in edge scenarios with limited hardware resources.

[0047] According to an embodiment of the present invention, before operation S1 identifies the first event pulse corresponding to the static background in the dynamic vision sensor and performs a removal process to obtain the second event pulse, the method further includes:

[0048] Use the inertial measurement unit to obtain the acceleration of the dynamic vision sensor;

[0049] The acceleration is integrated to obtain the velocity information of the dynamic vision sensor.

[0050] In this embodiment, for example, the motion information of the entire sensor is first obtained based on the inertial measurement unit (IMU), that is, the inertial sensor data (the inertial measurement unit and the dynamic vision sensor are kept fixed together). The accelerometer included in the IMU provides the acceleration of the sensor, and then the speed information (motion speed) of the sensor is obtained by integrating the acceleration.

[0051] According to an embodiment of the present invention, operation S1 identifies a first event pulse corresponding to a static background in a dynamic vision sensor and performs a removal process to obtain a second event pulse, which may specifically include:

[0052] Compressing the event pulses of the dynamic visual sensor into the same sequence, wherein the pulse frequency of the same sequence is positively correlated with the moving speed of the target;

[0053] The compressed event pulse sequence is input into the spiking neuron model for pulse filtering, wherein the activation threshold of the spiking neuron model is determined by the speed information;

[0054] Based on the activation threshold, the first event pulse below the activation threshold is eliminated to obtain the second event pulse above the activation threshold.

[0055] Dynamic vision sensor pulse compression assumes that event pulses in a small spatial region over a short period of time are all generated by the motion of the same object. Therefore, in this embodiment, these pulses are first compressed into a common sequence whose pulse frequency is positively correlated with the object's motion speed. This compressed pulse sequence is then input into a common spiking neuron model (e.g., the LIF neuron model).

[0056] The LIF neuron model (Leaky Integrate-and-Fire model) is a model applied to spiking neural networks, aiming to simulate the working process of biological neurons.

[0057] Secondly, set the activation threshold of the LIF neuron model according to the velocity information given by the IMU, so as to filter out the pulses with a frequency less than a specific emission frequency, obtain the pulses with velocities s > s1 and s > s2 in the scene, and then eliminate the pulses with velocities satisfying s1 < s < s2 in the scene, and finally achieve the elimination of the event pulses (the first event pulses) generated by the relative motion of the static background, and obtain the filtered pulses (the second event pulses).

[0058] The dynamic target tracking and recognition method based on the fusion of dynamic vision and single-photon sensors provided by the embodiments of the present invention simply realizes the elimination of pulses with a specific frequency through spiking neurons, that is, it distinguishes static objects and dynamic objects in the scene. The event pulses generated by the relative motion of static objects are useless for the intelligent vision task of target tracking and recognition. By eliminating them, the amount of data to be processed subsequently is greatly reduced.

[0059] According to an embodiment of the present invention, operation S2 inputs the second event pulses into the lightweight attention prediction spiking neural network in the form of a sequence for prediction, and obtains an attention heat map, which specifically may include:

[0060] Perform position encoding on the second event pulses;

[0061] Perform temporal encoding on the second event pulses after position encoding;

[0062] Use the self-attention mechanism to calculate the key-value pair attention between the sequences of the second event pulses after temporal encoding;

[0063] According to the key-value pair attention, use a multi-layer perceptron to predict the attention heat map at the current moment.

[0064] In this embodiment, first, perform learnable position encoding on the second event pulses obtained after filtering. For example, use a learnable encoding matrix to expand the encoding of the input event sequence with a length of n. Among them, the specific encoding expansion method can be: <​​​​​​​​​​​​​Secondly, the second event pulse after the position coding is time-coded, for example, by using the timestamp of the event to distinguish the order of events in the event sequence. Specifically, the second event pulse after the position coding can be time-coded using sine functions and cosine functions of different frequencies. The time coding method can be:

[0068]

[0069]

[0070] Where TE represents the time code and t represents the timestamp.

[0071] Furthermore, the above position code and event code are added together, and a self-attention calculation module is used, for example, a self-attention mechanism is used to calculate the key-value pair attention between the encoded sequences.

[0072] Finally, the above results are input into a LIF neuron model array of size n×d. The memory characteristics of the LIF neuron model are used to process the real-time time series information of the DVS data. For example, a multi-layer perceptron is used, using the output of the LIF neuron model as input to output the attention prediction result at the current moment.

[0073] The dynamic target tracking and recognition method of dynamic vision-single-photon sensor fusion provided by an embodiment of the present invention, in terms of the processing method of event pulse sequences, directly processes the sequences by an attention prediction pulse neural network, compared with the traditional processing method of accumulating sequences into frames, maintaining the sparsity of information while also retaining the temporal sequence information, and processing the sequence sequence information by neurons with memory characteristics, thereby having lower computational complexity while maintaining higher network performance.

[0074] According to an embodiment of the present invention, operation S3 crops the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map, which may specifically include:

[0075] generating a confidence matrix, wherein the confidence matrix has the same size as a pixel of the single-photon sensor;

[0076] Improve the confidence of the area near the confidence matrix according to the distribution characteristics of the attention heat map;

[0077] In response to the confidence reaching a confidence threshold, activating the single photon sensor and the corresponding convolutional neural network;

[0078] The attention heat map is processed by clustering algorithm to obtain the attention distribution coordinates;

[0079] The grayscale image of the single-photon sensor is cropped according to the attention distribution coordinates.

[0080] In this embodiment, a confidence matrix with the same size as the SPAD pixel is first generated, and then the confidence of the nearby area is improved according to the distribution of the attention heat map.

[0081] When the confidence reaches the confidence threshold, the SPAD and the corresponding convolutional neural network (CNN network) are activated, and then the attention heat map is processed by the clustering algorithm to obtain the main attention distribution coordinates, and finally the SPAD grayscale image is cropped.

[0082] The dynamic target tracking and recognition method using dynamic vision and single-photon sensor fusion provided in an embodiment of the present invention uses a lightweight small network with smaller parameters and faster speed to control a convolutional neural network with larger parameters and slower speed, so that the convolutional neural network no longer remains in a working state for a long time, thereby reducing the overall power consumption of the system.

[0083] Figure 2 The structural block diagram of a dynamic target tracking and recognition system using dynamic vision and single-photon sensor fusion according to an embodiment of the present invention is schematically shown.

[0084] like Figure 2 As shown, the structure of the dynamic target tracking and recognition system of the dynamic vision-single photon sensor fusion of this embodiment may include, for example: a background pulse removal module, an attention prediction pulse neural network module (SNN attention network), an attention-based grayscale image downsampling module, a target recognition convolutional neural network module (CNN recognition network) and a fusion module.

[0085] Among them, the background pulse removal module is used to identify the first event pulse corresponding to the static background in the dynamic vision sensor and perform removal processing to obtain the second event pulse, wherein the first event pulse is generated by the relative motion between the static background and the dynamic vision sensor.

[0086] In this embodiment, the background pulse rejection module uses the velocity information provided by the IMU sensor to filter out pulses through LIF neurons, and performs static background recognition on DVS dynamic flow events to eliminate event pulses caused by the relative motion between the static background and the sensor when the sensor is in motion.

[0087] The attention prediction pulse neural network module is used to input the second event pulse in the form of a sequence into the lightweight attention prediction pulse neural network for prediction to obtain an attention heat map.

[0088] In this embodiment, the attention prediction pulse neural network module may specifically include: a learnable position encoding module, a calculation module using self-attention, a LIF pulse neuron array module, and an attention prediction module, such as Figure 3 shown.

[0089] Figure 3 The structural block diagram of the attention prediction pulse neural network module according to an embodiment of the present invention is schematically shown.

[0090] like Figure 3 As shown, in the attention prediction pulse neural network module of this embodiment:

[0091] The learnable position encoding module uses a 2×d trainable matrix to achieve the expansion from n×2 pulse sequences to n×d sequences.

[0092] The calculation module using self-attention introduces the self-attention mechanism to calculate the key-value pair attention of the encoded sequence.

[0093] The LIF pulse neuron array module uses an n×d LIF neuron array and utilizes the memory characteristics of LIF neurons to process the real-time timing information of DVS data.

[0094] The attention prediction module uses a multi-layer perceptron and uses the output of the LIF spike neuron array module as input. It contains three fully connected layers, each of which includes but is not limited to: a 2×2 maximum pooling layer and an activation layer ReLU, such as Figure 4 shown.

[0095] Figure 4 The figure schematically shows the input and output effect diagram of the attention prediction pulse neural network module according to an embodiment of the present invention.

[0096] like Figure 4 As shown, the DVS sensor data in the car driving scene is input (left picture), where the car in front is the object that needs attention, and the attention prediction pulse neural network module can output the system's attention prediction result at the current moment (right picture).

[0097] The attention-based grayscale image downsampling module is used to crop the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map.

[0098] The target recognition convolutional neural network module is used to input the cropped grayscale image into the convolutional neural network for recognition to obtain the recognition result.

[0099] In this embodiment, the attention-based grayscale image downsampling module and the target recognition convolutional neural network module are both controlled by the output of the attention prediction pulse neural network module. The former downsamples and crops the grayscale image output by the SPAD according to the attention prediction results, and the latter controls the switch by the real-time attention prediction results, and uses the downsampled and cropped grayscale image as input to achieve target recognition.

[0100] The fusion module is used to fuse the attention heat map and recognition results to obtain the target tracking and recognition results.

[0101] Figure 5 The structural block diagram of an electronic device suitable for implementing a dynamic target tracking and recognition method of dynamic vision-single photon sensor fusion according to an embodiment of the present invention is schematically shown. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0102] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0103] The storage unit 508 stores various programs and data required for the operation of the electronic device 500. The processor 501, ROM 502, and storage unit 508 are interconnected via a bus 504. The processor 501 executes the programs stored in the ROM 502 and / or storage unit 508 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and storage unit 508. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0104] According to an embodiment of the present invention, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0105] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0106] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0107] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0108] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or the storage portion 508 described above and / or one or more memories other than the ROM 502 and the storage portion 508 .

[0109] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0110] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0111] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0112] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0114] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A dynamic target tracking and recognition method based on dynamic vision and single-photon sensor fusion, characterized in that: The method comprises: Identifying a first event pulse corresponding to a static background in a dynamic vision sensor and performing a removal process to obtain a second event pulse, wherein the first event pulse is generated by relative motion between the static background and the dynamic vision sensor; Inputting the second event pulse in the form of a sequence into a lightweight attention prediction pulse neural network for prediction to obtain an attention heat map; Cropping the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map; The cropped grayscale image is input into the convolutional neural network for recognition to obtain the recognition result; The attention heat map and the recognition result are integrated to obtain the target tracking and recognition result.

2. The method according to claim 1, characterized in that The process of identifying the first event pulse corresponding to the static background in the dynamic vision sensor and performing a removal process to obtain the second event pulse includes: Obtaining the acceleration of the dynamic vision sensor using an inertial measurement unit; The acceleration is integrated to obtain velocity information of the dynamic vision sensor.

3. The method according to claim 2, characterized in that The identifying of the first event pulse corresponding to the static background in the dynamic vision sensor and performing a removal process to obtain the second event pulse includes: compressing event pulses of the dynamic vision sensor into a same sequence, wherein the pulse frequency of the same sequence is positively correlated with the movement speed of the target; Inputting the compressed event pulse sequence into a spiking neuron model for pulse filtering, wherein the activation threshold of the spiking neuron model is determined by the speed information; Based on the activation threshold, first event pulses below the activation threshold are eliminated to obtain second event pulses above the activation threshold.

4. The method according to claim 1, wherein Inputting the second event pulse in a sequence into a lightweight attention prediction pulse neural network for prediction to obtain an attention heat map includes: performing position encoding on the second event pulse; performing time encoding on the second event pulse after the position encoding; Using the self-attention mechanism, the key-value pair attention between the second event pulse sequence after time encoding is calculated; Based on the key-value pair attention, a multi-layer perceptron is used to predict the attention heat map at the current moment.

5. The method according to claim 4, characterized in that The position encoding of the second event pulse comprises: Using a learnable coding matrix, the second event pulse sequence of input length n is coded and expanded, wherein the coding expansion method includes: Among them, π t represents the result after the event pulse sequence is expanded, x t represents the event pulse sequence, W p Represents a learnable encoding matrix, R represents a real number, and d represents the expanded dimension.

6. The method according to claim 4, characterized in that The performing time coding on the position-coded second event pulse comprises: The second event pulse after position encoding is time-encoded using sine functions and cosine functions of different frequencies, wherein the time encoding method includes: Where TE represents the time code and t represents the timestamp.

7. The method according to claim 1, characterized in that The cropping of the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map includes: generating a confidence matrix, wherein the confidence matrix has the same size as a pixel of the single photon sensor; Improving the confidence of an area near the confidence matrix according to the distribution characteristics of the attention heat map; In response to the confidence reaching a confidence threshold, activating the single photon sensor and the corresponding convolutional neural network; Processing the attention heat map through a clustering algorithm to obtain attention distribution coordinates; The grayscale image of the single-photon sensor is cropped according to the attention distribution coordinates.

8. A dynamic target tracking and recognition system based on dynamic vision and single-photon sensor fusion, characterized in that: The system comprises: a background pulse removal module, configured to identify a first event pulse corresponding to a static background in the dynamic vision sensor and remove the first event pulse to obtain a second event pulse, wherein the first event pulse is generated by relative motion between the static background and the dynamic vision sensor; an attention prediction pulse neural network module, configured to input the second event pulse in a sequence into a lightweight attention prediction pulse neural network for prediction, thereby obtaining an attention heat map; An attention-based grayscale image downsampling module, configured to crop the grayscale image of the single-photon sensor according to the distribution characteristics of the attention heat map; The target recognition convolutional neural network module is used to input the cropped grayscale image into the convolutional neural network for recognition and obtain the recognition result; A fusion module is used to fuse the attention heat map and the recognition result to obtain the target tracking and recognition result.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images