Model optimization method, device and electronic equipment based on pulse neural network

Abstract

The present invention provides a model optimization method, device and electronic device based on a pulse neural network, belonging to the field of computer technology. The method includes: modifying at least part of the network layer in the artificial neural network model to obtain a modified artificial neural network model; replacing the activation function of the modified artificial neural network model with a target neuron model to generate an initial pulse neural network model, the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step; jointly tuning the initial membrane potential and the initial threshold voltage of the target neuron model so that the tuned initial pulse neural network model reaches the optimal accuracy, and obtaining a target pulse neural network model, which is used to be deployed on an unmanned mobile platform. The model optimization method based on a pulse neural network provided by the present invention can complete the conversion of the artificial neural network model with low latency and low precision loss.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model optimization method, device and electronic equipment based on a pulse neural network. Background Art

[0002] At present, pulse neural networks are the main research object in the field of deep learning and brain-like computing, and are called "third-generation neural networks." Models based on this type of network have not yet been widely used on unmanned mobile platforms with independent computing capabilities such as drones or robots. The main reason is that the current pulse neural network model training, conversion, and operation occupy too much video memory or memory, while unmanned mobile platforms often have only extremely limited edge computing capabilities, and most of them do not have graphics cards or high memory, and cannot bear such high computing requirements. Most universal pulse neural network models require extremely high memory to complete operating tasks, which creates a contradiction between the advancedness and practicality of the pulse neural network model, making it impossible to play a role in unmanned mobile platforms in actual scenarios. Summary of the invention

[0003] The present invention provides a model optimization method, device and electronic equipment based on a pulse neural network, which are used to solve the problem in the prior art that the pulse neural network model cannot play a role in an unmanned mobile platform.

[0004] The present invention provides a model optimization method based on a pulse neural network, comprising:

[0005] Modifying at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0006] Replacing the activation function of the modified artificial neural network model with the target neuron model to generate an initial spiking neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0007] The initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model are jointly tuned so that the tuned initial pulse neural network model reaches the optimal accuracy, and the target pulse neural network model is obtained. The target pulse neural network model is used to be deployed on an unmanned mobile platform.

[0008] The method of modifying at least part of the network layers in the artificial neural network model includes at least one of the following:

[0009] When at least part of the network layers are multiple convolutional layers of the same dimension, and the number of consecutive stacks of the multiple convolutional layers of the same dimension exceeds a first threshold, adding residual blocks to the multiple convolutional layers of the same dimension;

[0010] In a case where at least some of the network layers are multiple parallel convolutional layers, reducing the multiple convolutional layers to a single convolutional layer;

[0011] In a case where at least part of the network layers are multiple parallel pooling layers, replacing the multiple pooling layers with a single convolutional layer;

[0012] In a case where the at least part of the network layers are spatial pyramid pooling layers, the spatial pyramid pooling layers are replaced with a single convolutional layer.

[0013] The jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model includes:

[0014] Simultaneously adjusting the initial membrane potential and the initial threshold voltage, and successively calculating the accuracy loss of the tuned initial pulse neural network model;

[0015] When the accuracy loss of the tuned initial pulse neural network model reaches a minimum, stop adjusting the initial membrane potential and the initial threshold voltage.

[0016] In some embodiments, the simultaneously adjusting the initial membrane potential and the initial threshold voltage comprises:

[0017] When the initial membrane potential is increased, the initial threshold voltage is increased or decreased.

[0018] In some embodiments, before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, the method further includes:

[0019] Determining the initial threshold voltage based on a preset pulse emission threshold of the target neuron model;

[0020] The initial membrane potential is determined based on the resting potential of the target neuron model.

[0021] In some embodiments, before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, the method further includes:

[0022] Setting the initial membrane potential of the target neuron model to 0V;

[0023] The initial threshold voltage of the target neuron model is set to 1V.

[0024] The present invention also provides a model optimization device based on a pulse neural network, comprising:

[0025] A first processing module is used to modify at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0026] A second processing module is used to replace the activation function of the modified artificial neural network model with a target neuron model to generate an initial pulse neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0027] A tuning module is used to jointly tune the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model so that the tuned initial pulse neural network model reaches the optimal accuracy, and the target pulse neural network model is obtained. The target pulse neural network model is used to be deployed on an unmanned mobile platform.

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a model optimization method based on a pulse neural network as described above is implemented.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described model optimization methods based on pulse neural networks.

[0030] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the model optimization method based on the pulse neural network as described above is implemented.

[0031] The model optimization method, device and electronic device based on the pulse neural network provided by the present invention can complete the conversion of the artificial neural network model to the pulse neural network model with low delay and low precision loss by modifying at least part of the network layers in the artificial neural network model and replacing the activation function, and at the same time can ensure that the target pulse neural network model with optimal precision is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a flow chart of the model optimization method based on pulse neural network provided by the present invention;

[0034] Figure 2 It is a schematic diagram of the time domain separation mechanism of the model optimization method based on the pulse neural network provided by the present invention;

[0035] Figure 3 It is one of the schematic diagrams of model performance parameters of the model optimization method based on pulse neural network provided by the present invention;

[0036] Figure 4 This is the second schematic diagram of model performance parameters of the model optimization method based on the pulse neural network provided by the present invention;

[0037] Figure 5 This is the third schematic diagram of model performance parameters of the model optimization method based on the pulse neural network provided by the present invention;

[0038] Figure 6 It is a structural schematic diagram of a model optimization device based on a pulse neural network provided by the present invention;

[0039] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Combine the following Figure 1-Figure 7 The present invention describes a pulse neural network-based model optimization method, device and electronic device.

[0042] The execution subject of the model optimization method based on pulse neural network provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited by the present invention.

[0043] The following takes the computer executing the pulse neural network-based model optimization method provided by the present invention as an example to explain the technical solution of the present invention in detail.

[0044] Figure 1 Schematic diagram of the flow of the model optimization method based on the pulse neural network provided by the present invention. Figure 1 The model optimization method based on pulse neural network provided by the present invention includes: step 110, step 120 and step 130.

[0045] Step 110: modify at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0046] Step 120: replacing the activation function of the modified artificial neural network model with the target neuron model to generate an initial pulse neural network model, where the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0047] Step 130, jointly tune the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model so that the tuned initial pulse neural network model reaches the optimal accuracy, and obtain the target pulse neural network model, which is used to be deployed on an unmanned mobile platform.

[0048] It should be noted that the Artificial Neural Network (ANN) model is a mathematical model that uses a structure similar to the synaptic connections in the brain to process information.

[0049] The artificial neural network model in the present invention can be selected from a high-performance dedicated computing platform, the artificial neural network model is the model before conversion, and the initial pulse neural network model is the model after conversion.

[0050] The following is a detailed description of the process of converting the artificial neural network model into the initial spiking neural network model.

[0051] In step 110, at least part of the network layers in the artificial neural network model are modified to better adapt to the pulse data. The modification method includes but is not limited to replacing, reducing and adding part of the layer structure in the model, so as to obtain a modified artificial neural network model.

[0052] The purpose of modifying the artificial neural network model is to reduce the error generated in the conversion process under the premise of limited impact on the accuracy of the artificial neural network model, so as to ensure that the impact on the model accuracy is small while reducing the model memory.

[0053] In actual implementation, the modified artificial neural network model can be trained to determine the accuracy loss of the model before and after the modification, and to judge whether the accuracy has dropped too much. If the accuracy loss is too large, it can be modified again until the modified artificial neural network model is the optimal model. The optimal model is the model with reduced model complexity and minimal accuracy loss.

[0054] In step 120, a target neuron model is selected. The target neuron model is a device used to simulate neuron state changes, such as an IF (Integrate-and-Fire) neuron, a LIF (Leaky Integrate-and-Fire) neuron, or an iLIF (inverse-Leaky Integrited-and-Fire) neuron.

[0055] In this step, the activation function of the modified artificial network model is replaced with the target neuron model, and a time domain separation mechanism is added to the target neuron model to generate an initial spiking neural network model. Using a neuron model with a time domain separation mechanism can reduce the delay of the initial spiking neural network.

[0056] The activation function may be a rectified linear unit (ReLU) activation function, or a sigmoid activation function or a softmax activation function, etc., which is not specifically limited here.

[0057] The time domain separation mechanism separates the accumulation and release of the membrane potential of the target neuron model into two stages within a time step, so that the membrane potential can be accumulated and released in a concentrated manner within the set time step. The use of the target neuron model enables the modified artificial network model to receive and process pulse train signals.

[0058] In actual implementation, Figure 2 As shown in Figure 1, each time step is divided into two processes, including a time-domain separation accumulation phase and a time-domain separation release phase. In the time-domain separation accumulation phase, the target neuron model only accumulates membrane potential and does not emit pulses; in the time-domain separation release phase, it only emits pulses and does not accumulate membrane potential.

[0059] In some embodiments, after the target neuron model is replaced, the weight parameters and bias parameters may be adjusted, and a batch normalization (BN) layer may be used to perform normalization and fusion operations to generate an initial pulse neural network model.

[0060] In step 130, the generated initial spiking neural network model is cross-tested and optimized based on the initial threshold voltage and the initial membrane potential to obtain an optimized initial spiking neural network model.

[0061] It should be noted that the initial threshold voltage and the initial membrane potential need to be tuned jointly. If one value is adjusted alone, the adjustment of the other value will no longer work after the conversion accuracy error of the pulse neural network model after model conversion reaches the minimum, and the tuned initial pulse neural network model cannot achieve the optimal accuracy.

[0062] When the initial pulse neural network model after tuning reaches the optimal accuracy, the tuning process is stopped to obtain the target pulse neural network model. The target pulse neural network model can continue to be specially trained to have the ability to detect a single type of target, thereby completing the task of building a target pulse neural network model with low precision loss and low latency.

[0063] The trained target pulse neural network model can then be deployed to unmanned mobile platforms with independent computing capabilities, such as drones, robots or unmanned vehicles based on central processing units (CPUs), graphics processing units (GPUs) or neuromorphic computing chips.

[0064] For example, the trained model is deployed to the neuromorphic processor of an unmanned mobile platform with limited performance (e.g., a drone or mobile robot) for real-machine testing. The indicators that affect the performance of the unmanned mobile platform include the calculation speed and frame rate, and the target loss situation can be judged. The specific deployment method and calculation indicators will be described in detail in the subsequent embodiments.

[0065] The model optimization method based on pulse neural network provided by the present invention can complete the conversion of artificial neural network model to pulse neural network model with low delay and low precision loss by modifying at least part of the network layers in the artificial neural network model and replacing the activation function, and at the same time can ensure that the target pulse neural network model with optimal precision is obtained.

[0066] In some embodiments, the method of modifying at least part of the network layers in the artificial neural network model includes at least one of the following:

[0067] When at least part of the network layers are multiple convolutional layers of the same dimension, and the number of consecutive stacks of the multiple convolutional layers of the same dimension exceeds a first threshold, adding residual blocks to the multiple convolutional layers of the same dimension;

[0068] In the case where at least some of the network layers are multiple convolutional layers in parallel, reducing the multiple convolutional layers to a single convolutional layer;

[0069] In the case where at least some of the network layers are multiple pooling layers in parallel, the multiple pooling layers are replaced by a single convolutional layer;

[0070] In a case where the at least part of the network layers are spatial pyramid pooling layers, the spatial pyramid pooling layers are replaced with a single convolutional layer.

[0071] In actual implementation, for complex artificial neural network models, multi-layer networks that are too deep or too wide will result in low model accuracy.

[0072] At least part of the network layers that are too deep or too wide are modified, and the model loss of the artificial neural network model before and after the modification is small.

[0073] The following is an explanation of how to modify the artificial neural network model.

[0074] For at least part of the network layer that is too deep, if at least part of the network layer is a plurality of convolutional layers of the same dimension, and the number of consecutive stacks of the plurality of convolutional layers of the same dimension exceeds a first threshold, a residual block is added to the plurality of convolutional layers of the same dimension to optimize the error after conversion. The first threshold can be determined according to actual needs, for example, it can be set to 3 or more, which is not specifically limited here.

[0075] For at least some network layers that are too wide, that is, there are multiple convolutional layers in parallel, multiple pooling layers in parallel, or Spatial Pyramid Pooling-Fast (SPPF) layers in the model, multiple convolutional layers can be reduced to a single convolutional layer, or multiple pooling layers can be reduced to a single pooling layer, or multiple pooling layers or Spatial Pyramid Pooling-Fast layers can be replaced with a single convolutional layer.

[0076] If the modified artificial neural network model produces too much precision loss, consider re-modifying the model to minimize the precision loss and achieve model conversion with less precision loss. For example, you can retain multiple parallel convolutional layers or multiple parallel pooling layers.

[0077] In some embodiments, jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model includes:

[0078] The initial membrane potential and the initial threshold voltage are adjusted simultaneously, and the accuracy loss of the optimized initial spike neural network model is calculated one by one;

[0079] When the accuracy loss of the tuned initial spike neural network model reaches a minimum, stop adjusting the initial membrane potential and the initial threshold voltage.

[0080] Under the premise of using time-domain separated neurons, the values ​​of the initial membrane potential and the initial threshold voltage are adjusted simultaneously to find the membrane potential and the threshold voltage when the accuracy loss is minimized.

[0081] In actual implementation, the initial membrane potential and the initial threshold voltage are adjusted at the same time. Each time an adjustment is made, the precision loss is calculated, and the combination of membrane potential and threshold voltage corresponding to the minimum precision loss is selected as the final value.

[0082] In some embodiments, before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, the method further includes:

[0083] Determining an initial threshold voltage based on a preset pulse firing threshold of a target neuron model;

[0084] Based on the resting potential of the target neuron model, the initial membrane potential is determined.

[0085] In actual implementation, based on the preset pulse emission threshold of the target neuron model and the resting potential of the target neuron model, the preset pulse emission threshold is used as the initial threshold voltage, and the resting potential is used as the initial membrane potential. The preset pulse emission threshold can be the default pulse emission threshold of the neuron, or the pulse emission threshold determined according to actual needs, which is not specifically limited here.

[0086] In some embodiments, before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, the method further includes:

[0087] Set the initial membrane potential of the target neuron model to 0V;

[0088] Set the initial threshold voltage of the target neuron model to 1V.

[0089] In actual implementation, the initial threshold voltage is set to 1 V. Optionally, the initial membrane potential can also be set to 0.5V.

[0090] In some embodiments, adjusting the initial membrane potential and the initial threshold voltage simultaneously comprises:

[0091] When the initial membrane potential is increased, the initial threshold voltage is increased or decreased.

[0092] In actual implementation, the maximum initial threshold voltage adjustment gradient is 0.1%, and the maximum initial membrane potential adjustment gradient is 1%; the threshold voltage value range is (0, 1.1], and when the initial threshold voltage is 1V, the initial threshold voltage can be increased or decreased based on this value range.

[0093] The present invention also provides a method process and evaluation scheme for adjusting and deploying a high-performance brain-like computing model on an unmanned mobile platform, the main process of which includes:

[0094] 1. Complete the construction, conversion, training and tuning of the spiking neural network model on the high-performance computing platform to obtain the target spiking neural network model; and complete the deployment, debugging and performance testing of the target spiking neural network model on the unmanned mobile platform.

[0095] 2. Conduct operational evaluation on the target spiking neural network model.

[0096] In actual implementation, the artificial neural network model is first designed, optimized and trained on a high-performance dedicated computing platform, and then the model is modified and converted from the analog value operation module to the pulse operation module in the model.

[0097] The analog value operation module refers to at least part of the network layer in the artificial neural network model, and the pulse operation module refers to the target neuron model. The use of the pulse operation module enables the converted artificial neural network model to receive and process pulse sequence signals.

[0098] After the conversion is completed, the generated spiking neural network model is subjected to cross-test tuning based on the reduction of the initial threshold voltage and the increase of the initial membrane potential.

[0099] During the model deployment phase, the following process is performed:

[0100] 1. Prepare at least one of the following devices or modules:

[0101] A developable unmanned mobile platform with independent computing capabilities and meeting the computing power requirements of the task; a neuromorphic processor; a spiking neural network model; a spiking neural network decoding module; a model operation index detection module; a related input signal capture module (for example, including: various sensors such as cameras, depth cameras or microphones); an input signal encoding module.

[0102] 2. Deployment Method

[0103] 1. Device initialization

[0104] Use a mobile hard disk, wireless network or wired network to transfer the pulse neural network model to an unmanned mobile platform and configure the model running environment. The unmanned mobile platform can be a robot or a drone.

[0105] When using an unmanned mobile platform running the Robot Operating System (ROS), attention should be paid to the compatibility of the operating environment with the ROS system.

[0106] For example, for a visual model written in python3, the CV2 module of the model's operating environment should be pointed to the CV2 module in the OpenCV package, otherwise incompatibility will occur. For multimodal tasks or tasks that require feedback from mechanical parts such as robotic arms or wheels, multiple environments can be executed in parallel. Among them, the CV2 module is used to complete a variety of computer vision and image processing tasks.

[0107] Then run the robot operating system. During the operation, you can start RViz as a monitoring auxiliary program on the remote or local end.

[0108] 2. Module assembly

[0109] An input signal encoding module is provided to encode signals received by different sensors;

[0110] Set batch parameters to determine the number of data in each batch;

[0111] Normalize the analog value signal corresponding to each batch of data. For example, the image data needs to be reduced by 255 times; then set the time step timesteps value (neuron state refresh rate), repeat the image data timesteps times, and send it to the pulse neural network model;

[0112] At the module output, a decoder module is provided to decode the pulse sequence;

[0113] Set up a model operation indicator detection module to provide necessary data for model evaluation. This module should include: memory usage detection or video memory usage detection, model inference speed detection, output frame rate detection, as well as model detection capability and error loss detection functions;

[0114] 3. Model operation

[0115] Run the model to confirm that it can run smoothly.

[0116] 3. Model Evaluation

[0117] 1. Run the model using the labeled dataset and run the indicator detection module;

[0118] 2. Check the accuracy data of the dataset, including map@0.5 and map@0.5:0.9 data, memory / video memory operation status and growth trend, and inference speed value;

[0119] 3. Use the model to detect real-time video, voice or other types of signal streams to check the actual target detection effect; check the single-frame inference speed and the actual return frame rate; check whether the returned detection results are updated and played smoothly;

[0120] 4. For models that require feedback from mechanical components, it is necessary to test the reliability of the actual component operation.

[0121] 5. The test time should be guaranteed to be maintained for more than half an hour under high-intensity information flow input to test the stability of memory usage. If memory explosion occurs, it is necessary to adjust the model data storage capacity and accuracy to improve memory usage.

[0122] 4. Model Evaluation Data Processing Method

[0123] 1. map@0.5 or map@0.5:0.9: This data is the general accuracy data of deep learning, which represents the detection rate and position overlap of the model detection results relative to the annotation content.

[0124] 2. mem_gpu: represents the GPU usage during model training.

[0125] 3. inference_time: represents the inference time of a single frame signal in the model, excluding the input signal encoding and non-maximum suppression (NMS) part;

[0126] 4. In the model provided by the present invention, the analysis time of each frame (preprocessing + reasoning + low-confidence target filtering) is also detected, and the output image size is represented.

[0127] 5. When the unmanned mobile platform moves at high speed, it is necessary to test the upper limit of the speed and distance that it can support to effectively detect targets. The specific standards are as follows:

[0128] (1) Maximum recognizable distance

[0129] like Figure 3 As shown in the figure, the maximum recognizable distance between the unmanned mobile platform and the target meets the following conditions:

[0130]

[0131] R represents the farthest recognizable distance, r represents the farthest camera shooting distance, and θ represents the maximum shooting angle of the camera; D min It represents the minimum recognizable object size, M represents the maximum width of the picture that the camera can sample, and the number of pixels can be used as a comparison here.

[0132] In actual situations, The model performance can be annotated as the maximum distance at which an object can be identified.

[0133] (2) Maximum lateral tracking speed

[0134] like Figure 4 As shown in the figure, the maximum lateral tracking speed of the unmanned mobile platform equipped with the model meets the following conditions:

[0135]

[0136] Where v represents the maximum recognizable (relative) running speed of the lateral target, R represents the maximum recognizable distance, and t t represents the single frame analysis and processing time, and θ represents the maximum shooting angle of the camera.

[0137] (3) Maximum target tracking relative speed

[0138] like Figure 5 As shown, the maximum target tracking relative speed satisfies the following relationship:

[0139]

[0140] Where v represents the maximum target tracking relative speed.

[0141] It should be noted that the above values ​​are the highest values ​​under ideal conditions, which may not be achieved due to the influence of the model and hardware sensors. Therefore, it is necessary to reduce certain quantities based on actual conditions. min It needs to be determined based on the actual capabilities of the model. At the same time, the frame rate fps = 1 / t t ,This frame rate is limited by the model inference speed and is also affected by the camera performance.

[0142] Through the above methods, the performance indicators of the pulse neural network model on the unmanned mobile platform can be measured, and the controllable variables can be adjusted in time according to actual needs to obtain a human mobile platform deployment that meets the requirements.

[0143] Compared with the prior art, the present invention proposes a transformation and deployment scheme that enables the pulse neural network model to run on an unmanned mobile platform, making the pulse neural network model practical on the unmanned mobile platform. At the same time, through the improvement of the tuning scheme, it takes a step towards a high-performance model. An evaluation method after the model is deployed on an unmanned mobile platform is proposed, so that the pulse neural network model has a quantifiable performance evaluation method in multiple scenarios.

[0144] The model optimization device based on the pulse neural network provided by the present invention is described below. The model optimization device based on the pulse neural network described below and the model optimization method based on the pulse neural network described above can refer to each other.

[0145] Figure 6 Schematic diagram of the structure of the model optimization device based on the pulse neural network provided by the present invention. Figure 6 The model optimization device based on the pulse neural network provided by the present invention comprises:

[0146] A first processing module 610 is used to modify at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0147] A second processing module 620 is used to replace the activation function of the modified artificial neural network model with a target neuron model to generate an initial spiking neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0148] The tuning module 630 is used to jointly tune the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model so that the tuned initial pulse neural network model reaches the optimal accuracy and obtains the target pulse neural network model, which is used to be deployed on an unmanned mobile platform.

[0149] The model optimization method, device and electronic device based on the pulse neural network provided by the present invention can complete the conversion of the artificial neural network model to the pulse neural network model with low delay and low precision loss by modifying at least part of the network layers in the artificial neural network model and replacing the activation function, and at the same time can ensure that the target pulse neural network model with optimal precision is obtained.

[0150] In some embodiments, the first processing module 610 is used for at least one of the following:

[0151] When at least part of the network layers are multiple convolutional layers of the same dimension, and the number of consecutive stacks of the multiple convolutional layers of the same dimension exceeds a first threshold, adding residual blocks to the multiple convolutional layers of the same dimension;

[0152] In a case where the at least part of the network layers are a plurality of parallel convolutional layers, reducing the plurality of convolutional layers to a single convolutional layer;

[0153] In the case where the at least part of the network layers are multiple parallel pooling layers, the multiple pooling layers are replaced with spatial pyramid pooling fast layers.

[0154] In some embodiments, the tuning module 630 is specifically used to:

[0155] Simultaneously adjusting the initial membrane potential and the initial threshold voltage, and successively calculating the accuracy loss of the tuned initial pulse neural network model;

[0156] When the accuracy loss of the tuned initial pulse neural network model reaches a minimum, stop adjusting the initial membrane potential and the initial threshold voltage.

[0157] In some embodiments, the tuning module 630 is specifically used to:

[0158] When the initial membrane potential is increased, the initial threshold voltage is increased or decreased.

[0159] In some embodiments, the apparatus further comprises:

[0160] An initialization module, used for determining the initial threshold voltage based on a preset pulse emission threshold of the target neuron model before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model;

[0161] The initial membrane potential is determined based on the resting potential of the target neuron model.

[0162] In some embodiments, the initialization module is further used to:

[0163] Setting the initial membrane potential of the target neuron model to 0V;

[0164] The initial threshold voltage of the target neuron model is set to 1V.

[0165] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a model optimization method based on a pulse neural network, and the method includes:

[0166] Modifying at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0167] Replacing the activation function of the modified artificial neural network model with the target neuron model to generate an initial spiking neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0168] The initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model are jointly tuned so that the tuned initial pulse neural network model reaches the optimal accuracy, and the target pulse neural network model is obtained. The target pulse neural network model is used to be deployed on an unmanned mobile platform.

[0169] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0170] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the model optimization method based on the pulse neural network provided by the above methods, the method includes:

[0171] Modifying at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0172] Replacing the activation function of the modified artificial neural network model with the target neuron model to generate an initial spiking neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0173] The initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model are jointly tuned so that the tuned initial pulse neural network model reaches the optimal accuracy, and the target pulse neural network model is obtained. The target pulse neural network model is used to be deployed on an unmanned mobile platform.

[0174] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the model optimization method based on a pulse neural network provided by the above methods, the method comprising:

[0175] Modifying at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model;

[0176] Replacing the activation function of the modified artificial neural network model with the target neuron model to generate an initial spiking neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step;

[0177] The initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model are jointly tuned so that the tuned initial pulse neural network model reaches the optimal accuracy, and the target pulse neural network model is obtained. The target pulse neural network model is used to be deployed on an unmanned mobile platform.

[0178] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model optimization method based on a pulse neural network, characterized in that: include: Modifying at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model; Replacing the activation function of the modified artificial neural network model with the target neuron model to generate an initial spiking neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step; The initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model are jointly tuned so that the tuned initial pulse neural network model reaches the optimal accuracy, and a target pulse neural network model is obtained, wherein the target pulse neural network model is used for deployment on an unmanned mobile platform; the input of the target pulse neural network model is image data; the target pulse neural network model is used for tracking identifiable targets; The method of modifying at least part of the network layers in the artificial neural network model includes at least one of the following: When at least part of the network layers are multiple convolutional layers of the same dimension, and the number of consecutive stacks of the multiple convolutional layers of the same dimension exceeds a first threshold, adding residual blocks to the multiple convolutional layers of the same dimension; In a case where at least some of the network layers are multiple parallel convolutional layers, reducing the multiple convolutional layers to a single convolutional layer; In a case where at least part of the network layers are multiple parallel pooling layers, replacing the multiple pooling layers with a single convolutional layer; In a case where the at least part of the network layers are spatial pyramid pooling layers, replacing the spatial pyramid pooling layers with a single convolutional layer; The jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model includes: Simultaneously adjusting the initial membrane potential and the initial threshold voltage, and successively calculating the accuracy loss of the tuned initial pulse neural network model; When the accuracy loss of the tuned initial pulse neural network model reaches a minimum, stop adjusting the initial membrane potential and the initial threshold voltage.

2. The model optimization method based on pulse neural network according to claim 1, characterized in that: The simultaneously adjusting the initial membrane potential and the initial threshold voltage comprises: When the initial membrane potential is increased, the initial threshold voltage is increased or decreased.

3. The model optimization method based on pulse neural network according to claim 1, characterized in that: Before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, the method further includes: Determining the initial threshold voltage based on a preset pulse emission threshold of the target neuron model; The initial membrane potential is determined based on the resting potential of the target neuron model.

4. The model optimization method based on pulse neural network according to any one of claims 1 to 3, characterized in that: Before jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, the method further includes: Setting the initial membrane potential of the target neuron model to 0V; The initial threshold voltage of the target neuron model is set to 1V.

5. A model optimization device based on a pulse neural network, characterized in that: include: The first processing module is used to modify at least part of the network layers in the artificial neural network model to obtain a modified artificial neural network model, including at least one of the following: When at least part of the network layers are multiple convolutional layers of the same dimension, and the number of consecutive stacks of the multiple convolutional layers of the same dimension exceeds a first threshold, adding residual blocks to the multiple convolutional layers of the same dimension; In a case where at least some of the network layers are multiple parallel convolutional layers, reducing the multiple convolutional layers to a single convolutional layer; In a case where at least part of the network layers are multiple parallel pooling layers, replacing the multiple pooling layers with a single convolutional layer; In a case where the at least part of the network layers are spatial pyramid pooling layers, replacing the spatial pyramid pooling layers with a single convolutional layer; A second processing module is used to replace the activation function of the modified artificial neural network model with a target neuron model to generate an initial pulse neural network model, wherein the target neuron model is used to complete the accumulation and release of the membrane potential of the target neuron model in each time step; A tuning module is used to jointly tune the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model, so that the tuned initial pulse neural network model reaches the optimal accuracy, and obtains a target pulse neural network model, wherein the target pulse neural network model is used to be deployed on an unmanned mobile platform; the input of the target pulse neural network model is image data; the target pulse neural network model is used to track identifiable targets; The jointly tuning the initial membrane potential of the target neuron model and the initial threshold voltage of the target neuron model includes: Simultaneously adjusting the initial membrane potential and the initial threshold voltage, and successively calculating the accuracy loss of the tuned initial pulse neural network model; When the accuracy loss of the tuned initial pulse neural network model reaches a minimum, stop adjusting the initial membrane potential and the initial threshold voltage.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the model optimization method based on the pulse neural network as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model optimization method based on a pulse neural network as claimed in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model optimization method based on a pulse neural network as claimed in any one of claims 1 to 4 is implemented.

Images