Visual tracking method, device and electronic device based on pulse neural network

Abstract

The present invention provides a visual tracking method, device and electronic device based on a pulse neural network, which belongs to the field of computer technology. The method includes: acquiring two adjacent frames of RGB images to generate a target event image; inputting the target event image and the latter RGB image of the two adjacent frames of RGB images into a multimodal tracking model for visual tracking; the multimodal tracking model includes a fusion module and a twin network model, the fusion module is used to fuse the target event image and the latter RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output the positioning result of the tracking target in the latter RGB image. The visual tracking method based on a pulse neural network provided by the present invention can improve the visual tracking capability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a visual tracking method, device and electronic equipment based on a pulse neural network. Background Art

[0002] At present, pulse neural networks are the main research object in the field of deep learning and brain-like computing, and are called "third-generation neural networks." Models based on this type of network have not yet been widely used on unmanned mobile platforms with independent computing capabilities, such as drones or robots. The main reason is that the current pulse neural network model has too large a delay, low precision, and occupies too much video memory or memory, while unmanned mobile platforms often have only extremely limited edge computing capabilities, resulting in low efficiency and poor results in completing tracking tasks, or even inability to complete tracking tasks. The low efficiency and poor results have caused a contradiction between the advancement and practicality of the pulse neural network model, making it impossible to demonstrate its advancement and practicality on unmanned platforms in actual scenarios. Summary of the invention

[0003] The present invention provides a pulse neural network-based visual tracking method, device and electronic equipment, which are used to solve the problem of low efficiency of pulse neural network models in completing tracking tasks on unmanned mobile platforms in the prior art.

[0004] In a first aspect, the present invention provides a visual tracking method based on a spiking neural network, comprising:

[0005] Get two adjacent RGB image frames and generate a target event image;

[0006] Inputting the target event image and the latter RGB image of the two adjacent RGB images into a multimodal tracking model for visual tracking;

[0007] The multimodal tracking model includes a fusion module and a twin network model, wherein the fusion module is used to fuse the target event image and the subsequent frame RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output a positioning result of the tracking target in the subsequent frame RGB image;

[0008] The activation layers of the fusion module and the twin network model in the multimodal tracking model are replaced with neuron models, the central cropping structure of the twin network model is replaced with a synaptic inhibition mechanism, and the multimodal tracking model is a type of pulse neural network model.

[0009] As an optional embodiment, the present invention further provides a visual tracking method based on a spiking neural network, wherein two adjacent frames of RGB images are acquired to generate a target event image:

[0010] Performing grayscale processing on the two adjacent frames of RGB images to obtain two adjacent frames of grayscale images;

[0011] The target event image is generated based on a pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale images, wherein the pixel difference is greater than a first difference threshold or less than a second difference threshold, wherein the first difference threshold is a positive value and the second difference threshold is a negative value.

[0012] As an optional embodiment, the present invention further provides a visual tracking method based on a spiking neural network, wherein the target event image is generated based on the pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale images, comprising:

[0013] Setting a pixel value corresponding to a first pixel position whose pixel difference is greater than the first difference threshold as a first preset value, and setting a pixel value of a second pixel position as 0, to obtain a first feature map, wherein the second pixel position is a pixel position other than the first pixel position among all pixel positions in the grayscale image;

[0014] The pixel value corresponding to the third pixel position whose pixel difference value is less than the second difference threshold is set as the second preset value, and the pixel value of the fourth pixel position is set as 0, to obtain a second feature map, wherein the fourth pixel position is the pixel position other than the third pixel position among all pixel positions in the grayscale image;

[0015] Determine a first pixel value at each pixel position based on the sum of pixel values ​​of two pixels at the same pixel position in the first feature map and the second feature map;

[0016] Subtract the pixel difference values ​​corresponding to the corresponding pixel positions in the two adjacent grayscale images from the first pixel value of each pixel position to obtain a second pixel value of each pixel position;

[0017] The target event image is generated based on the first difference threshold and the second pixel value of each pixel position.

[0018] As an optional embodiment, the present invention further provides a visual tracking method based on a spiking neural network, wherein the fusion module fuses the target event image and the subsequent frame RGB image to obtain a multimodal image, specifically comprising:

[0019] After performing a convolution operation on the target event image using a 3x3 convolution, performing size restoration using a deconvolution, and outputting a corresponding event feature map;

[0020] Splicing the RGB image and the event feature map to obtain a 4-channel feature map;

[0021] Perform a convolution operation on the 4-channel feature map using a 1x1 convolution to obtain a 3-channel feature map;

[0022] The 3-channel feature map is used as the multimodal image.

[0023] As an optional embodiment, the present invention further provides a visual tracking method based on a spiking neural network, wherein the neuron model includes an IF neuron, a LIF neuron or an iLIF neuron;

[0024] In the case where the activation layer of the fusion module and / or the twin network model is a rectified linear unit ReLU layer, replacing the rectified linear unit ReLU layer with an IF neuron;

[0025] When the activation layer of the fusion module and / or the twin network model is a leaky-rectified linear unit (Leaky-ReLU) layer, the leaky-rectified linear unit (Leaky-ReLU) layer is replaced with a LIF neuron or an iLIF neuron.

[0026] As an optional embodiment, the present invention also provides a visual tracking method based on a pulse neural network.

[0027] When the twin network model has the center cropping structure, neuron pulses outside the output size threshold of the neuron model are suppressed, and the center cropping structure is used to reduce the image size of the image input to the twin network model.

[0028] In a second aspect, the present invention further provides a visual tracking device based on a pulse neural network, comprising:

[0029] An image preprocessing unit, used to obtain two adjacent frames of RGB images and generate a target event image;

[0030] A visual tracking module, used to input the target event image and the latter RGB image of the two adjacent RGB images into a multimodal tracking model for visual tracking; the multimodal tracking model includes a fusion module and a twin network model, the fusion module is used to fuse the target event image and the latter RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output a positioning result of the tracking target in the latter RGB image;

[0031] The activation layers of the fusion module and the twin network model in the multimodal tracking model are replaced with neuron models, the central cropping structure of the twin network model is replaced with a synaptic inhibition mechanism, and the multimodal tracking model is a type of pulse neural network model.

[0032] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a visual tracking method based on a pulse neural network as described above is implemented.

[0033] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the pulse neural network-based visual tracking methods described above.

[0034] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described pulse neural network-based visual tracking methods.

[0035] The visual tracking method, device and electronic device based on pulse neural network provided by the present invention use two adjacent frames of RGB images to generate a target event image, use the fusion module in the multimodal tracking model to fuse the target event image and the current RGB image, and finally use the synaptic inhibition mechanism to convert the twin network model to obtain a pulse neural network model to recognize the fused multimodal image, which can improve the model operation speed and reduce energy consumption, so that the trained pulse neural network model can improve the tracking efficiency and tracking accuracy when performing visual tracking, and have more outstanding tracking capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is one of the flow charts of the visual tracking method based on pulse neural network provided by the present invention;

[0038] Figure 2 It is a schematic diagram of the inhibition processing of the visual tracking method based on the pulse neural network provided by the present invention;

[0039] Figure 3 This is the second flow chart of the visual tracking method based on the pulse neural network provided by the present invention;

[0040] Figure 4 It is a schematic diagram of the event image generation process of the visual tracking method based on the pulse neural network provided by the present invention;

[0041] Figure 5 It is a schematic diagram of image fusion of a visual tracking method based on a pulse neural network provided by the present invention;

[0042] Figure 6 It is one of the schematic diagrams of model performance parameters of the visual tracking method based on pulse neural network provided by the present invention;

[0043] Figure 7 This is the second schematic diagram of model performance parameters of the visual tracking method based on the pulse neural network provided by the present invention;

[0044] Figure 8 This is the third schematic diagram of model performance parameters of the visual tracking method based on the pulse neural network provided by the present invention;

[0045] Fig. 9 It is a schematic diagram of the structure of a visual tracking device based on a pulse neural network provided by the present invention;

[0046] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Combine the following Figure 1-Figure 10 The present invention describes a visual tracking method, device and electronic device based on a pulse neural network.

[0049] The execution subject of the pulse neural network-based visual tracking method provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited by the present invention.

[0050] The following takes the computer executing the pulse neural network-based visual tracking method provided by the present invention as an example to explain the technical solution of the present invention in detail.

[0051] Figure 1 Schematic diagram of the flow of the visual tracking method based on the pulse neural network provided by the present invention. Figure 1 The visual tracking method based on pulse neural network provided by the present invention includes: step 110 and step 120.

[0052] Step 110, acquiring two adjacent frames of RGB images to generate a target event image;

[0053] Step 120: Input the target event image and the latter RGB image of the two adjacent RGB images into a multimodal tracking model for visual tracking.

[0054] The multimodal tracking model includes a fusion module and a twin network model, wherein the fusion module is used to fuse the target event image and the subsequent frame RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output the positioning result of the tracking target in the subsequent frame RGB image;

[0055] The activation layers of the fusion module and the twin network model in the multimodal tracking model are replaced with neuron models, the central cropping structure of the twin network model is replaced with a synaptic inhibition mechanism, and the multimodal tracking model is a type of pulse neural network model.

[0056] It should be noted that the twin network model provided by the present invention is a coupling framework based on two artificial neural networks (ANN). The twin network model takes two samples as input and outputs their representation embedded in a high-dimensional space to compare the similarity of the two samples. The ANN model is a mathematical model that uses a structure similar to the synaptic connection of the brain to process information.

[0057] The following is an explanation of the process of converting the twin network model to the pulse neural network model.

[0058] In actual implementation, since the model construction and initialization stages, training and conversion work require higher video memory and memory, a twin network model is built on a high-performance dedicated computing platform, and the twin network model is trained to obtain a trained twin network model.

[0059] Then, the trained twin network model is converted to obtain a pulse neural network model. Specifically, a suitable neuron model is selected to replace the activation layer in the twin network model.

[0060] The neuron model is a device used to simulate the change of neuron state, for example, it can be an IF (Integrate-and-Fire) neuron, a LIF (Leaky Integrate-and-Fire) neuron or an iLIF (inverse-Leaky Integrited-and-Fire) neuron. The activation layer can be a Rectified Linear Unit (ReLU) layer or a Leaky-ReLU layer.

[0061] It is also possible to use a neuron model with a time domain separation mechanism to reduce the delay of the spiking neural network, thereby solving the high delay problem of the traditional spiking neural network model. Then, the weight parameters and bias parameters are standardized over the channel range. The initial pulse activation threshold voltage is set to 1, and the initial membrane potential is generally 0.5.

[0062] The time-domain separation mechanism separates the accumulation and release of the membrane potential of the neuron model into two stages within a time step, so that the membrane potential can be accumulated and released in a concentrated manner within the set time step. The use of the neuron model enables the modified artificial network model to receive and process pulse train signals. The neuron model can divide each time step into two processes, including the time-domain separation accumulation stage and the time-domain separation release stage. In the time-domain separation accumulation stage, the target neuron model only accumulates membrane potential and does not emit pulses; in the time-domain separation release stage, it only emits pulses and does not accumulate membrane potential.

[0063] After converting the twin network model into a pulse neural network model, the generated pulse neural network model can be tested and tuned based on the initial membrane potential and pulse activation threshold modification according to the model performance, and then the above process can be repeated.

[0064] The generated spiking neural network model is trained to obtain a trained spiking neural network model, and the trained spiking neural network model is used for visual tracking.

[0065] If the visual tracking target is specific, for example, mainly tracking a certain type of object, the gradient function or loss function can be modified and replaced, and the pulse neural network model can be trained using a small proprietary data set to achieve the relevant effect. If there is no relevant requirement, the conversion is completed.

[0066] It is understandable that the trained spiking neural network model can be deployed to unmanned mobile platforms with independent computing capabilities, such as drones, robots or unmanned vehicles based on central processing units (CPU), graphics processing units (GPU) or neuromorphic computing chips.

[0067] For example, the trained model is deployed to the neuromorphic processor of an unmanned mobile platform with limited performance (e.g., a drone or mobile robot) for real-machine testing. The indicators that affect the performance of the unmanned mobile platform include the calculation speed and frame rate, and the target loss situation can be judged. The specific deployment method and calculation indicators will be described in detail in the subsequent embodiments.

[0068] The visual tracking method based on a pulse neural network provided by the present invention, the visual tracking method based on a pulse neural network provided by the present invention, the device and the electronic device use two adjacent frames of RGB images to generate a target event image, use a fusion module in a multimodal tracking model to fuse the target event image and the current RGB image, and finally use a synaptic inhibition mechanism to convert the twin network model to obtain a pulse neural network model to recognize the fused multimodal image, which can improve the model operation speed and reduce energy consumption, thereby enabling the trained pulse neural network model to improve tracking efficiency and tracking accuracy during visual tracking and have more outstanding tracking capabilities.

[0069] In some embodiments, the neuron model comprises an IF neuron, a LIF neuron, or an iLIF neuron;

[0070] In the case where the activation layer of the fusion module and / or the twin network model is a rectified linear unit ReLU layer, replacing the rectified linear unit ReLU layer with an IF neuron;

[0071] When the activation layer of the fusion module and / or the twin network model is a leaky-rectified linear unit (Leaky-ReLU) layer, the leaky-rectified linear unit (Leaky-ReLU) layer is replaced with a LIF neuron or an iLIF neuron.

[0072] In some embodiments, after replacing the activation layer in the twin network model with a neuron model to generate a spiking neural network model, the method further includes:

[0073] In the presence of a target structure in the twin network model, neuronal pulses outside the output size threshold of the neuron model are suppressed, and the target structure is used to reduce the image size.

[0074] It should be noted that some twin network ANNs in the twin network model may use the target structure to reduce the size of the feature map. Therefore, it is necessary to convert the target structure into a "synaptic inhibition mechanism". The target structure can be the Center_Clip structure in the related art.

[0075] The synaptic inhibition mechanism refers to the equivalent mapping of the image size reduction process to the spiking neural network model. Figure 2 As shown, by suppressing the neuron pulse output outside the size threshold, the number of output neurons is reduced, that is, the image size is reduced. Figure 2 The area corresponding to the dotted box is the size of the size threshold.

[0076] It should be noted that when the size of the feature map received by the network layer containing the synaptic inhibition mechanism is larger than the size threshold, synaptic inhibition will be performed to reduce the image size.

[0077] The size threshold can be determined according to actual needs and is not specifically limited here.

[0078] In the field of visual tracking, event cameras are expensive and lack the ability to perceive static objects quickly. Event cameras have not yet been widely used. The development of low-cost event cameras or low-cost equivalent alternatives has become one of the main directions. At the same time, the fusion of RGB images and event images is considered to be one of the effective ways to make up for the disadvantages of event cameras.

[0079] In some embodiments, the Siamese network model is trained as follows:

[0080] Perform grayscale processing on two adjacent frames of RGB image samples to obtain two adjacent frames of grayscale image samples;

[0081] Generate a target event image sample based on a pixel difference between two pixels at the same pixel position in two adjacent frames of grayscale image samples, the pixel difference is greater than a first difference threshold or less than a second difference threshold, the first difference threshold is a positive value, and the second difference threshold is a negative value;

[0082] The target event image sample is fused with the two adjacent frames of RGB image samples to obtain the target image sample;

[0083] Based on the target image samples, the Siamese network model is trained.

[0084] In actual implementation, two adjacent frames of RGB image samples are obtained and converted into two adjacent frames of grayscale image samples, and the brightness change value is calculated at the pixel level, that is, the pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale image samples is calculated.

[0085] It is understandable that the pixel difference value can be a positive value or a negative value. A positive value indicates a positive change in brightness, that is, an increase in brightness; a negative value indicates a negative change in brightness, that is, a decrease in brightness.

[0086] The initial difference threshold may be set to 10, and the initial threshold includes a first difference threshold and a second difference threshold, and the first difference threshold is 10 and the second difference threshold is -10. The initial difference threshold may also be set to other values, which are not specifically limited here.

[0087] When the pixel difference is greater than the first difference threshold, it can be used to indicate a first characteristic map with a positive brightness change.

[0088] When the pixel difference is less than the second difference threshold, a second characteristic map may be used to indicate a negative change in brightness.

[0089] When the pixel difference is less than or equal to the first difference threshold and greater than or equal to the second difference threshold, it can indicate that the brightness change is small.

[0090] In this step, a target event image sample may be generated based on the first feature map and the second feature map, and then the target event image sample is fused with two adjacent frames of RGB image samples to obtain a target image sample.

[0091] The image fusion process provided by the present invention is as follows Figure 3 and Figure 5 shown.

[0092] Specifically, the fusion module fuses the target event image and the subsequent frame RGB image to obtain a multimodal image, which specifically includes:

[0093] After performing a convolution operation on the target event image using a 3x3 convolution, performing size restoration using a deconvolution, and outputting a corresponding event feature map;

[0094] Splicing the RGB image and the event feature map to obtain a 4-channel feature map;

[0095] Perform a convolution operation on the 4-channel feature map using a 1x1 convolution to obtain a 3-channel feature map;

[0096] The 3-channel feature map is used as the multimodal image.

[0097] The visual tracking method based on pulse neural network provided by the present invention generates target event images based on RGB image samples, which greatly reduces the acquisition cost of target event image samples.

[0098] In some embodiments, generating the target event image based on the pixel difference between two pixels at the same pixel position in the two adjacent grayscale image frames includes:

[0099] Setting a pixel value corresponding to a first pixel position whose pixel difference is greater than the first difference threshold as a first preset value, and setting a pixel value of a second pixel position as 0, to obtain a first feature map, wherein the second pixel position is a pixel position other than the first pixel position among all pixel positions in the grayscale image;

[0100] The pixel value corresponding to the third pixel position whose pixel difference value is less than the second difference threshold is set as the second preset value, and the pixel value of the fourth pixel position is set as 0, to obtain a second feature map, wherein the fourth pixel position is the pixel position other than the third pixel position among all pixel positions in the grayscale image;

[0101] Determine a first pixel value at each pixel position based on the sum of pixel values ​​of two pixels at the same pixel position in the first feature map and the second feature map;

[0102] Subtract the pixel difference values ​​corresponding to the corresponding pixel positions in the two adjacent grayscale images from the first pixel value of each pixel position to obtain a second pixel value of each pixel position;

[0103] The target event image is generated based on the first difference threshold and the second pixel value of each pixel position.

[0104] In actual implementation, after determining the pixel difference between two pixel points corresponding to each pixel position in two adjacent frames of grayscale image samples, the pixel value corresponding to the first pixel position whose pixel difference is greater than the first difference threshold is set as the first preset value, and the pixel value of the second pixel position is set to 0, to obtain a first feature map, that is, a positive feature map;

[0105] The pixel value corresponding to the third pixel position whose pixel difference is less than the second difference threshold is set to the second preset value, and the pixel value of the fourth pixel position is set to 0, to obtain a second feature map, that is, a negative feature map.

[0106] In actual implementation, a positive change sets the pixel value corresponding to the pixel position to 255, and a negative change sets the pixel value corresponding to the pixel position to -255, that is, the first preset value is 255, and the second preset value is -255.

[0107] Based on the pixel values ​​of all pixel positions in the first feature map or the second feature map, two arrays can be formed, the two arrays are added at the pixel level, and then the pixel difference between the corresponding pixel positions of the two frames of grayscale images is subtracted to form a new array.

[0108] A target event image sample is generated according to the first difference threshold and the second pixel value of each pixel position indicated by the new array.

[0109] In actual implementation, Figure 4 As shown, RGB image 1 and RGB image 2 are two adjacent RGB images. After grayscale processing of the two adjacent RGB images, grayscale image 1 and grayscale image 2 are obtained, and the pixel difference diff of each pixel position in grayscale image 1 and grayscale image 2 is calculated. At the pixel position where the pixel difference diff is greater than the first difference threshold 10, the pixel value is defined as 255, and a positive feature map (pos) can be generated, or at the position where the pixel difference diff is less than the second difference threshold -10, the pixel value is defined as -255, and a negative feature map (neg) can be generated. 255 is the first preset value, and -255 is the second preset value.

[0110] As an optional embodiment, the step of acquiring two adjacent RGB image frames to generate a target event image is as follows:

[0111] Performing grayscale processing on the two adjacent frames of RGB images to obtain two adjacent frames of grayscale images;

[0112] The target event image is generated based on a pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale images, wherein the pixel difference is greater than a first difference threshold or less than a second difference threshold, wherein the first difference threshold is a positive value and the second difference threshold is a negative value.

[0113] In some embodiments, generating a target event image sample based on the first difference threshold and the second pixel value of each pixel position includes:

[0114] Increasing the pixel value corresponding to the pixel position where the second pixel value exceeds the first difference threshold, and determining the pixel position corresponding to the increased pixel value;

[0115] Lowering the pixel value corresponding to the pixel position where the second pixel value does not exceed the first difference threshold, and determining the pixel position corresponding to the lowered pixel value;

[0116] A target event image sample is generated based on the pixel position corresponding to the increased pixel value and the pixel position corresponding to the decreased pixel value.

[0117] In the generated new array, the elements exceeding the first difference threshold are highlighted, that is, the pixel value is increased; the elements that do not exceed the brightness change threshold are reduced in brightness, that is, reduced to, for example, 0. At this point, the target event image sample is generated. Observe the display of the event image for dynamic events, that is, observe the noise in the event image. If there are too many noise points and the motion part cannot be effectively extracted, increase the first difference threshold, otherwise reduce the first difference threshold. Then repeat the above process until the generation effect is ideal.

[0118] The visual tracking method based on pulse neural network provided by the present invention can reduce the noise in the target event image samples and optimize the generation effect of the event image, so that the twin network model trained by the target event image samples can perform more accurate visual tracking.

[0119] The following describes in detail the process of deploying the trained spiking neural network model to an unmanned mobile platform.

[0120] 1. Prepare at least one of the following devices or modules:

[0121] Mobile edge computing devices such as unmanned vehicles, robots, and drones, pulse neural network models for visual tracking, output decoding modules of pulse neural network models, operation test modules of pulse neural network models, input devices (RGB cameras), and input signal receiving and encoding modules.

[0122] 2. Deployment Method

[0123] 1. Device initialization

[0124] Step 1: Perform format conversion and other processing on the trained spiking neural network model, process it into a file format supported by the corresponding computing part of the unmanned mobile platform, and then deploy the code to the corresponding computing module of the unmanned mobile platform. For example, if a neuromorphic computing chip is used, the spiking neural network model should first be converted to a model format supported by the neuromorphic computing chip, and then deployed in a targeted manner according to the model requirements. For models with decoders, the decoder should also be deployed to the unmanned mobile platform.

[0125] Step 2: Check the compatibility of the unmanned mobile platform and the model compilation environment, and use Docker to transfer the environment if necessary.

[0126] Step 3: Configure the model running environment on the unmanned mobile platform. You can use the installation package to install or the unmanned mobile platform to automatically download and install it online. After the installation is complete, run the program to check the support level of the unmanned mobile platform computing architecture for different environment packages. If you encounter an unsupported environment package, you can add the corresponding patch function software package. If you use docker for environment configuration, skip this step and go directly to step 4.

[0127] Step 4: Modify the environment package that conflicts with the unmanned platform operating system. This can be achieved by modifying the call pointer, replacing the equivalent function package, etc. In actual execution, an unmanned mobile platform running the Robot Operating System (ROS) can be used. Attention should be paid to the compatibility of the operating environment and the ROS system, such as the compatibility of the programming language and the ROS system. For tasks that do not involve perception components and do not require action feedback from mechanical components such as robotic arms and wheels, the task parts that conflict with the ROS system environment can be executed through a parallel multi-environment method.

[0128] Step 5: Disable or limit the motion function of the unmanned platform, start the program, test run the program, observe the program running status through the remote tool SSH, the 3D reconstruction visualization platform (Reconstruct Visualization Interface, RVIZ) or the direct platform video signal, and debug the program until it can run normally.

[0129] Step 6: Run the model using the test data set as input to check whether the model runs normally. If it runs normally, go to step 7; if it fails, go to step 5.

[0130] Step 7: Update the input end to the data format or path supported by the sensor to obtain input data that can be parsed by the model encoding module; if data preprocessing is required, add a data preprocessing module.

[0131] Step 8: Run the model and check the tracking function. Use the test module to perform performance tests, with the main indicators including frame rate, success rate or accuracy. Check whether the loss of each indicator is too large. If the frame rate loss is too large, consider improving the hardware performance; if the success rate and accuracy are reduced too much, consider reinitializing the model.

[0132] Step 9: For unmanned mobile platforms that use ROS as the operating system, you can use RVIZ as the monitoring program, or use Virtual Network Computing (VNC) remote desktop for control. Unmanned mobile platforms with voice control functions can be controlled by voice. For devices with wireless routing, you can consider networked remote operation and monitoring.

[0133] 3. Model Testing Process

[0134] 1. Use the labeled dataset as input to test the model performance after deployment on the unmanned mobile platform;

[0135] 2. Check various indicators, including Success data and Precision data, the operation and growth trend of memory or video memory, and the model inference speed; if the gap is too large, return and reinitialize the model.

[0136] 3. Change the model input to real-time signal and check the changes in visual tracking ability;

[0137] 4. For tasks that require mechanical feedback from the unmanned mobile platform, observe the accuracy of the response of its mechanical components. For example, in the visual tracking task of a robot or drone, the unmanned mobile platform needs to move accordingly to the target movement. If there is a large deviation, the mechanical component response program needs to be readjusted to ensure stability and reliability.

[0138] 5. The testing time should be sufficient to avoid performance degradation of the model after long-term running.

[0139] IV. Model Evaluation Reference Indicators

[0140] 1. Success or Precision, which is used to characterize the success rate and accuracy of the model tracking results relative to the annotated content.

[0141] 2. Fps, used to characterize the model inference speed.

[0142] 3. The measurement indicators of the maximum tracking capability of the unmanned mobile platform include: the longest recognizable distance, the maximum lateral tracking speed and the maximum target tracking relative speed.

[0143] (1) Maximum recognizable distance

[0144] like Figure 6 As shown in the figure, the maximum recognizable distance between the unmanned mobile platform and the target meets the following conditions:

[0145]

[0146] R represents the farthest recognizable distance, r represents the farthest camera distance, θ represents the maximum shooting angle of the camera, and D min Represents the minimum recognizable object size, and M represents the maximum width of the image that the camera can sample.

[0147] Returns the screen size and the minimum recognizable size of a specific target. The number of pixels can be used as a comparison here. In actual situations, this value can be used as the maximum distance at which the target can be recognized to mark the model performance.

[0148] (2) Maximum lateral tracking speed

[0149] like Figure 7 As shown in the figure, the maximum lateral tracking speed of the unmanned mobile platform equipped with the model meets the following conditions:

[0150]

[0151] Where v represents the maximum recognizable (relative) running speed of the lateral target, R represents the maximum recognizable distance, and t t represents the single frame analysis and processing time, and θ represents the maximum shooting angle of the camera.

[0152] (3) Maximum target tracking relative speed

[0153] like Figure 8 As shown, the maximum target tracking relative speed satisfies the following relationship:

[0154]

[0155] Where v represents the maximum target tracking relative speed.

[0156] It should be noted that the above values ​​are the highest values ​​under ideal conditions, which may not be achieved due to the influence of the model and hardware sensors. Therefore, it is necessary to reduce certain quantities based on actual conditions. min It needs to be determined based on the actual capabilities of the model. At the same time, the frame rate fps = 1 / t t ,This frame rate is limited by the model inference speed and is also affected by the camera performance.

[0157] At this point, the testing process ends. You can also adjust data variables and optimize the model and its deployment according to task requirements.

[0158] Compared with the prior art, the present invention proposes a method that can realize event image output within two frames using a low-cost RGB camera, and fuse RGB images with event images. It also proposes a method that is compatible with RGB images and event images and inputs them into a twin network model, and converts the twin network model into a pulse neural network model. It also provides a deployment and testing scheme for its operation on an unmanned mobile platform, so that the model has practicality on unmanned mobile platforms such as drones or robots, and has a quantifiable performance evaluation method in multiple scenarios.

[0159] As an optional embodiment, when the center cropping structure exists in the twin network model, neuron pulses outside the output size threshold of the neuron model are suppressed, and the center cropping structure is used to reduce the image size of the image input to the twin network model.

[0160] The visual tracking device based on the pulse neural network provided by the present invention is described below. The visual tracking device based on the pulse neural network described below and the visual tracking method based on the pulse neural network described above can be referenced to each other.

[0161] Fig. 9 Schematic diagram of the structure of the visual tracking device based on the pulse neural network provided by the present invention. Fig. 9 The visual tracking device based on the pulse neural network provided by the present invention comprises:

[0162] An image preprocessing unit 910 is used to obtain two adjacent frames of RGB images and generate a target event image;

[0163] The visual tracking module 920 is used to input the target event image and the latter RGB image of the two adjacent RGB images into a multimodal tracking model for visual tracking.

[0164] The multimodal tracking model includes a fusion module and a twin network model, wherein the fusion module is used to fuse the target event image and the subsequent frame RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output the positioning result of the tracking target in the subsequent frame RGB image;

[0165] The activation layers of the fusion module and the twin network model in the multimodal tracking model are replaced with neuron models, the central cropping structure of the twin network model is replaced with a synaptic inhibition mechanism, and the multimodal tracking model is a type of pulse neural network model.

[0166] The visual tracking device based on a pulse neural network provided by the present invention, the visual tracking method, device and electronic device based on a pulse neural network provided by the present invention use two adjacent frames of RGB images to generate a target event image, use a fusion module in a multimodal tracking model to fuse the target event image and the current RGB image, and finally use a synaptic inhibition mechanism to convert the twin network model to obtain a pulse neural network model to recognize the fused multimodal image, which can improve the model operation speed and reduce energy consumption, thereby enabling the trained pulse neural network model to improve tracking efficiency and tracking accuracy during visual tracking and have more outstanding tracking capabilities.

[0167] In some embodiments, the neuron model includes IF neurons, LIF neurons or iLIF neurons, specifically:

[0168] In the case where the activation layer is a rectified linear unit ReLU layer, the rectified linear unit ReLU layer is replaced by an IF neuron;

[0169] In the case where the activation layer is a leaky rectified linear unit (Leaky-ReLU) layer, the leaky rectified linear unit (Leaky-ReLU) layer is replaced by a LIF neuron or an iLIF neuron.

[0170] In some embodiments, the apparatus further comprises:

[0171] The suppression processing module is used for replacing the activation layer in the twin network model with a neuron model to generate a pulse neural network model. When there is a target structure in the twin network model, the neuron pulses outside the output size threshold of the neuron model are suppressed, and the target structure is used to reduce the image size.

[0172] In some embodiments, the twin network model is trained in the following manner:

[0173] Performing grayscale processing on the two adjacent frames of RGB image samples to obtain two adjacent frames of grayscale image samples;

[0174] Generate the target event image sample based on the pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale image samples, the pixel difference is greater than a first difference threshold or less than a second difference threshold, the first difference threshold is a positive value, and the second difference threshold is a negative value;

[0175] Fusing the target event image sample with the two adjacent frames of RGB image samples to obtain a target image sample;

[0176] Based on the target image sample, the twin network model is trained.

[0177] In some embodiments, the generating the target event image sample based on the pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale image samples includes:

[0178] Setting a pixel value corresponding to a first pixel position whose pixel difference is greater than the first difference threshold as a first preset value, and setting a pixel value at a second pixel position as 0, to obtain a first feature map, wherein the second pixel position is a pixel position other than the first pixel position among all pixel positions in the grayscale image sample;

[0179] The pixel value corresponding to the third pixel position whose pixel difference value is less than the second difference threshold is set as the second preset value, and the pixel value of the fourth pixel position is set as 0, to obtain a second feature map, wherein the fourth pixel position is the pixel position other than the third pixel position among all pixel positions in the grayscale image sample;

[0180] Determine a first pixel value at each pixel position based on the sum of pixel values ​​of two pixels at the same pixel position in the first feature map and the second feature map;

[0181] Subtract the pixel difference values ​​corresponding to the corresponding pixel positions in the two adjacent frames of grayscale image samples from the first pixel value of each pixel position to obtain a second pixel value of each pixel position;

[0182] The target event image sample is generated based on the first difference threshold and the second pixel value of each pixel position.

[0183] In some embodiments, generating the target event image sample based on the first difference threshold and the second pixel value of each pixel position includes:

[0184] Increasing the pixel value corresponding to the pixel position where the second pixel value exceeds the first difference threshold, and determining the pixel position corresponding to the increased pixel value;

[0185] Lowering the pixel value corresponding to the pixel position where the second pixel value does not exceed the first difference threshold, and determining the pixel position corresponding to the lowered pixel value;

[0186] The target event image sample is generated based on the pixel position corresponding to the increased pixel value and the pixel position corresponding to the decreased pixel value.

[0187] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute a visual tracking method based on a pulse neural network, and the method includes:

[0188] Replacing the activation layer in the twin network model with a neuron model, generating a pulse neural network model, and training the pulse neural network model to obtain a trained pulse neural network model, wherein the twin network model is trained based on a target event image sample and two adjacent frames of RGB image samples, and the target event image sample is determined based on the two adjacent frames of RGB image samples;

[0189] Visual tracking is performed based on the trained spiking neural network model.

[0190] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0191] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the visual tracking method based on the pulse neural network provided by the above methods, the method includes:

[0192] Replacing the activation layer in the twin network model with a neuron model, generating a pulse neural network model, and training the pulse neural network model to obtain a trained pulse neural network model, wherein the twin network model is trained based on a target event image sample and two adjacent frames of RGB image samples, and the target event image sample is determined based on the two adjacent frames of RGB image samples;

[0193] Visual tracking is performed based on the trained spiking neural network model.

[0194] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the visual tracking method based on a pulse neural network provided by the above methods, the method comprising:

[0195] Replacing the activation layer in the twin network model with a neuron model, generating a pulse neural network model, and training the pulse neural network model to obtain a trained pulse neural network model, wherein the twin network model is trained based on a target event image sample and two adjacent frames of RGB image samples, and the target event image sample is determined based on the two adjacent frames of RGB image samples;

[0196] Visual tracking is performed based on the trained spiking neural network model.

[0197] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0198] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual tracking method based on a pulse neural network, characterized in that: include: Get two adjacent RGB image frames and generate a target event image; Inputting the target event image and the latter RGB image of the two adjacent RGB images into a multimodal tracking model for visual tracking; The multimodal tracking model includes a fusion module and a twin network model, wherein the fusion module is used to fuse the target event image and the subsequent frame RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output a positioning result of the tracking target in the subsequent frame RGB image; The activation layers of the fusion module and the twin network model in the multimodal tracking model are replaced with neuron models, the central clipping structure of the twin network model is replaced with a synaptic inhibition mechanism, and the multimodal tracking model is a type of pulse neural network model; The synaptic inhibition mechanism is to map the image size reduction process equivalently to the pulse neural network model. When the twin network model has the central cropping structure, the neuron pulses outside the output size threshold of the neuron model are suppressed. The central cropping structure is used to reduce the image size input to the twin network model.

2. The visual tracking method based on pulse neural network according to claim 1, characterized in that: The method obtains two adjacent RGB image frames and generates a target event image: Performing grayscale processing on the two adjacent frames of RGB images to obtain two adjacent frames of grayscale images; The target event image is generated based on a pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale images, wherein the pixel difference is greater than a first difference threshold or less than a second difference threshold, wherein the first difference threshold is a positive value and the second difference threshold is a negative value.

3. The visual tracking method based on pulse neural network according to claim 2, characterized in that: The generating of the target event image based on the pixel difference between two pixels at the same pixel position in the two adjacent frames of grayscale images includes: Setting a pixel value corresponding to a first pixel position whose pixel difference is greater than the first difference threshold as a first preset value, and setting a pixel value of a second pixel position as 0, to obtain a first feature map, wherein the second pixel position is a pixel position other than the first pixel position among all pixel positions in the grayscale image; The pixel value corresponding to the third pixel position whose pixel difference value is less than the second difference threshold is set as the second preset value, and the pixel value of the fourth pixel position is set as 0, to obtain a second feature map, wherein the fourth pixel position is the pixel position other than the third pixel position among all pixel positions in the grayscale image; Determine a first pixel value at each pixel position based on the sum of pixel values ​​of two pixels at the same pixel position in the first feature map and the second feature map; Subtract the pixel difference values ​​corresponding to the corresponding pixel positions in the two adjacent grayscale images from the first pixel value of each pixel position to obtain a second pixel value of each pixel position; The target event image is generated based on the first difference threshold and the second pixel value of each pixel position.

4. The visual tracking method based on pulse neural network according to claim 1, characterized in that: The fusion module fuses the target event image and the subsequent frame RGB image to obtain a multimodal image, specifically including: After performing a convolution operation on the target event image using a 3x3 convolution, performing size restoration using a deconvolution, and outputting a corresponding event feature map; Splicing the RGB image and the event feature map to obtain a 4-channel feature map; Perform a convolution operation on the 4-channel feature map using a 1x1 convolution to obtain a 3-channel feature map; The 3-channel feature map is used as the multimodal image.

5. The visual tracking method based on pulse neural network according to claim 1, characterized in that: The neuron model includes IF neurons, LIF neurons or iLIF neurons; In the case where the activation layer of the fusion module and / or the twin network model is a rectified linear unit ReLU layer, replacing the rectified linear unit ReLU layer with an IF neuron; When the activation layer of the fusion module and / or the twin network model is a leaky-rectified linear unit (Leaky-ReLU) layer, the leaky-rectified linear unit (Leaky-ReLU) layer is replaced with a LIF neuron or an iLIF neuron.

6. A visual tracking device based on a pulse neural network, characterized in that: include: An image preprocessing unit, used to obtain two adjacent frames of RGB images and generate a target event image; A visual tracking module, used to input the target event image and the latter RGB image of the two adjacent RGB images into a multimodal tracking model for visual tracking; the multimodal tracking model includes a fusion module and a twin network model, the fusion module is used to fuse the target event image and the latter RGB image to obtain a multimodal image, and the twin network model is used to receive the multimodal image and output a positioning result of the tracking target in the latter RGB image; The activation layers of the fusion module and the twin network model in the multimodal tracking model are replaced with neuron models, the central clipping structure of the twin network model is replaced with a synaptic inhibition mechanism, and the multimodal tracking model is a type of pulse neural network model; The synaptic inhibition mechanism is to map the image size reduction process equivalently to the pulse neural network model. When the twin network model has the central cropping structure, the neuron pulses outside the output size threshold of the neuron model are suppressed. The central cropping structure is used to reduce the image size input to the twin network model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the visual tracking method based on the pulse neural network as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual tracking method based on a pulse neural network as claimed in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the visual tracking method based on a pulse neural network as claimed in any one of claims 1 to 5 is implemented.

Images