Event recognition model training method, device, apparatus, and event recognition method

Abstract

The present application relates to event recognition technology, in particular to an event recognition model training method, device and event recognition method, comprising: obtaining an event segment, video frame and label of event sample data; inputting the event segment and video frame into a pulse convolutional neural network and a memory Transformer network of an event recognition model to be trained respectively to obtain event segment features and video frame features; fusing the event segment features and video frame features to obtain a recognition result; adjusting model parameters according to a loss function between the recognition result and the label to obtain a trained event recognition model. The event recognition model trained by the present application uses a memory Transformer network to process RGB video, better representing time and space information of the RGB video; on the other hand, uses a pulse convolutional neural network to process an event stream, retaining high performance while reducing overhead; finally, fuses the dual-modal data to further improve recognition performance.

Description

TECHNICAL FIELD

[0001] The present application relates to event recognition technology, in particular to an event recognition model training method and device, an event recognition method and equipment. BACKGROUND

[0002] Event cameras have a wider application scenario and become a popular research field due to their low power consumption, low delay, high dynamic range and other advantages over traditional RGB cameras. Event stream recognition based on event cameras is a basic problem in this field. In order to realize event stream recognition, a feature representation based on event stream is first needed, and then an event recognition result is obtained by using a recognition model.

[0003] In addition, the effect of obtaining an event recognition result based only on an event stream is not ideal. Event cameras that can record dual-mode data of RGB video and event stream have appeared, such as DAVIS (Dynamic and Active Pixel Vision Sensor) 346 cameras, and it is of great significance to effectively utilize dual-mode data to achieve more efficient event recognition.

[0004] The existing event recognition method has the following problems:

[0005] (1) The existing event stream representation method of events will introduce redundant information or lose the spatiotemporal properties of the event stream;

[0006] (2) The artificial neural network (ANN) in the recognition model needs intensive calculation, which brings huge overhead, and the performance of the spiking neural network (SNN) is not high;

[0007] (3) The existing method cannot well utilize the dual-mode data of RGB video and event stream. SUMMARY

[0008] In view of the above-mentioned shortcomings of the prior art, the present application provides an event recognition model training method that can preserve the time and space information of events and improve the recognition performance by utilizing dual-mode data.

[0009] To achieve the above object and other related objects, the present application provides an event recognition model training method, comprising:

[0010] obtaining an event segment, a video frame and a label of event sample data;

[0011] inputting the event segment into a spiking convolutional neural network of an event recognition model to be trained to obtain an event segment feature;

[0012] input the video frame into a memory Transformer network of an event recognition model to be trained to obtain a video frame feature;

[0013] fuse the event segment feature and the video frame feature to obtain a recognition result;

[0014] adjust model parameters according to a loss function between the recognition result and the label to obtain an event recognition model after training.

[0015] In an embodiment of the present application, the step of obtaining the event segment, the video frame and the label of the event sample data comprises:

[0016] obtain the event sample data and the label;

[0017] process the event stream and the RGB video of the event sample data to obtain the event segment and the video frame.

[0018] In an embodiment of the present application, the step of inputting the event segment into a pulse convolutional neural network of an event recognition model to be trained to obtain an event segment feature comprises:

[0019] input the event segment into the pulse convolutional neural network to obtain a first feature, a second feature and a third feature;

[0020] splice the first feature, the second feature and the third feature and fuse them through a deformable convolutional neural network to obtain the event segment feature.

[0021] In an embodiment of the present application, the step of inputting the event segment into the pulse convolutional neural network to obtain a first feature, a second feature and a third feature comprises:

[0022] the event segment is processed eight times through a neural network with pulse neurons and two-dimensional convolution layers staggered, and the first feature, the second feature and the third feature are outputted after the eighth time, the sixth time and the fourth time of processing, respectively.

[0023] In an embodiment of the present application, the step of splicing the first feature, the second feature and the third feature and fusing them through a deformable convolutional neural network to obtain the event segment feature comprises:

[0024] splice the first feature after processing through two layers of residual networks and one layer of deconvolution layers with the second feature to obtain a first intermediate feature;

[0025] splice the first intermediate feature after processing through one layer of deconvolution layers with the third feature to obtain a second intermediate feature;

[0026] The second intermediate feature is processed through the deformable convolutional neural network to obtain the event segment feature.

[0027] In an embodiment of the present application, the step of inputting the video frame into the memory Transformer network of the event recognition model to be trained to obtain a video frame feature comprises:

[0028] The video frame is divided into a plurality of sequentially arranged video segments according to time;

[0029] The memory information corresponding to each video segment and the previous video segment is sequentially input into the memory Transformer network to obtain the corresponding intermediate video feature and the memory information;

[0030] The intermediate video feature output by the memory Transformer network of the last video segment is taken as the video frame feature.

[0031] In an embodiment of the present application, the step of fusing the event segment feature and the video frame feature to obtain an identification result comprises:

[0032] A plurality of event segment features and video frame features are spliced into event stream features and RGB video features, respectively;

[0033] According to the channel dimension, the event stream features and the RGB video features are spliced and fused to obtain the identification result.

[0034] To achieve the above object and other related objects, the present application further provides an event recognition method, comprising:

[0035] Obtaining event data to be recognized;

[0036] The event data is input into the event recognition model obtained by the training method of the event recognition model to obtain an event recognition result.

[0037] To achieve the above object and other related objects, the present application further provides an event recognition model training system, comprising:

[0038] An acquisition module is configured to acquire an event segment, a video frame and a label of event sample data;

[0039] An event segment feature acquisition module is configured to input the event segment into a pulse convolutional neural network of an event recognition model to be trained to obtain an event segment feature;

[0040] A video frame feature acquisition module is configured to input the video frame into a memory Transformer network of an event recognition model to be trained to obtain a video frame feature;

[0041] An obtaining module is configured to obtain the event segment features and the video frame features.

[0042] A model adjusting module is configured to adjust model parameters according to a loss function between the recognition result and the label to obtain a trained event recognition model.

[0043] To achieve the above object and other related objects, the present application further provides an electronic device, which comprises:

[0044] One or more processors;

[0045] A storage device is configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the training method of the event recognition model.

[0046] The present application obtains event stream features and RGB video features by processing the event stream and the RGB video of the event sample data through the pulse convolutional neural network and the memory Transformer network of the event recognition model to be trained respectively, and obtains a recognition result by fusing the two pieces of data in a double-modal manner, and finally adjusts the model parameters in combination with the label of the event sample data, so that the finally trained event recognition model not only has information of time and space at the same time, but also is accurate and high-performance. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application. It is readily apparent to one of ordinary skill in the art that the accompanying drawings are merely some embodiments of the present application, and other drawings can be obtained from the accompanying drawings without any creative effort. In the drawings:

[0048] Figure 1 is a flowchart of the training method of the event recognition model provided by the embodiments of the present application;

[0049] Figure 2 is a specific flowchart of the training method of the event recognition model provided by the embodiments of the present application;

[0050] Figure 3 is a flowchart of the memory Transformer network provided by the embodiments of the present application;

[0051] Figure 4 is a flowchart of the event recognition method provided by the embodiments of the present application;

[0052] Figure 5is a flowchart of a training system of an event recognition model provided by an embodiment of the present application;

[0053] Figure 6 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application can be implemented or applied in other different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application.

[0055] The Memory Transformer Network is different from the traditional Convolutional Neural Network (CNN) and the Recurrent Neural Networks (RNN) which can only be calculated from left to right or from right to left. The Memory Transformer is a combination of full connection and attention mechanism. The Memory Transformer can reduce the distance between any two positions in the sequence to a constant. It has better parallelism than the sequential structure of RNN, which conforms to the existing Graphics Processing Unit (GPU) framework.

[0056] Please refer to Figures 1-6 It should be noted that the diagrams provided in the embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be randomly changed in terms of shape, number and proportion, and the layout pattern of the components may be more complex.

[0057] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] Figure 1 A flowchart of a preferred embodiment of a training method of an event recognition model of the present application is shown,Figure 2 This is a flowchart illustrating the training method for the event recognition model provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the memory Transformer network provided in an embodiment of the present invention.

[0059] The following will combine Figure 1 , Figure 2 and Figure 3 The training method of the event recognition model of the present invention will be described in detail below, the method comprising:

[0060] Step S11: Obtain the event segments, video frames, and tags of the event sample data.

[0061] In one embodiment, the step includes: acquiring the event sample data and the tag; processing the event stream and RGB video of the event sample data to obtain the event segment and the video frame.

[0062] Specifically, the event recognition model to be trained in this invention is based on an event camera capable of simultaneously acquiring RGB video and event streams, and performs event recognition by combining the event stream of event sample data and RGB video. Event sample data with known results (i.e., labels) is acquired. The event stream size of the event sample data is T×X×Y×P, where T represents time, X represents the horizontal axis, Y represents the vertical axis, and P represents the polarity of the event. Spatial locations with events are represented as 1, and locations without events are marked as 0. The data is divided according to time t, forming segments of size t×x×y, totaling [number missing]. Paragraph. For example... Figure 2 As shown, the event stream of DVS is preprocessed and divided into several segments in chronological order to obtain an event segment containing n segments, and video frames are extracted from the corresponding event segments in the RGB video.

[0063] Step S12: Input the event segment into the pulse convolutional neural network of the event recognition model to be trained to obtain the event segment features.

[0064] In one embodiment, the step includes: inputting the event segment into the pulse convolutional neural network to obtain a first feature, a second feature, and a third feature; concatenating the first feature, the second feature, and the third feature and fusing them through a deformable convolutional neural network to obtain the event segment feature.

[0065] In one embodiment, the step of inputting the event segment into the spiking convolutional neural network to obtain the first feature, the second feature, and the third feature includes: the event segment is processed eight times by a neural network consisting of spiking neurons and two-dimensional convolutional layers, and the first feature, the second feature, and the third feature are output after the eighth, sixth, and fourth processing, respectively.

[0066] In an embodiment, the step of splicing the first feature, the second feature and the third feature and fusing through the deformable convolutional neural network to obtain the event segment feature comprises: splicing the first feature after being processed through two layers of residual network and one layer of deconvolution layer with the second feature to obtain a first intermediate feature; splicing the first intermediate feature after being processed through one layer of deconvolution layer with the third feature to obtain a second intermediate feature; processing the second intermediate feature through the deformable convolutional neural network to obtain the event segment feature.

[0067] Specifically, the Spiking Convolutional Neural Network (SCNN) of the event recognition model to be trained includes a convolution layer and a spiking Leaky Integrate-and-Fire (LIF) neuron model. The number n of segments contained in the event segment is taken as a simulation step, and the event segment is processed through an eight-layer two-dimensional convolution layer and a LIF neuron interlaced network structure, and the specific formula is

[0068] M i+1 =Lif(ConvM i ), 0≤i≤7

[0069] wherein M represents a feature output by a two-dimensional convolution and a LIF neuron, i represents a layer number of the two-dimensional convolution and the LIF neuron, Conv is a two-dimensional convolution layer, and Lif is a LIF neuron.

[0070] As Figure 2 indicated in the Simulation Step part, the pulse convolution is sequentially performed on the segments from t0 to t n , and the features are superimposed, and finally the pulse convolution is performed on the segment t n , and the processing results, i.e., a first feature A1, a second feature A2 and a third feature A3, are output after the eighth time, the sixth time and the fourth time of processing through the two-dimensional convolution layer and the LIF neuron interlaced, respectively, A1, A2 and A3 all include n segments, i.e., the features of the event segment. After A1 is processed through two layers of two-dimensional convolution with residual and one layer of deconvolution layer T1, A1 is spliced with A2 along the channel dimension to obtain a first intermediate feature; after the first intermediate feature is processed through one layer of deconvolution layer T2, the first intermediate feature is spliced with A3 along the channel dimension to obtain a second intermediate feature; and the second intermediate feature is processed through a deformable convolution module to obtain a fused event segment feature.

[0071] Step S13: inputting the video frame into a memory Transformer network of the event recognition model to be trained to obtain a video frame feature.

[0072] In an embodiment, the step comprises: dividing the video frame into a plurality of sequentially arranged video segments in time; inputting the memory information corresponding to each video segment and the previous video segment in sequence into the memory Transformer network to obtain the corresponding intermediate video feature and the memory information; and taking the intermediate video feature output by the memory Transformer network for the last video segment as the video frame feature.

[0073] Specifically, when a video frame is input into the memory Transformer network of the event recognition model to be trained, the video frame is divided into a plurality of sequentially arranged video segments in time. When the memory Transformer network processes the current video segment, the memory information of the previous video segment is combined, and the memory information of the current video segment is compressed and saved for use when the next video segment is processed while the intermediate video feature is output. The intermediate video feature output when the last video segment is processed is the video frame feature. As shown in Figure 2 In this embodiment, the video frame is divided into four video segments in time sequence. CLIP-1, CLIP-2, CLIP-3 and CLIP-4 in the figure correspond to the processing modules of the four video segments respectively. Each video segment is processed by three layers of MFormer network, and the memory information of the current processed video segment is compressed and saved by each layer of MFormer network. In this embodiment, K and V (i.e. key and value) in the attention mechanism are taken as the memory information. The first layer of MFormer network outputs the first intermediate video feature after processing the first video segment and compresses and saves the current memory information K11 and V11. The second layer of MFormer network outputs the second intermediate video feature after processing the first intermediate video feature and compresses and saves the current memory information K12 and V12. The third layer of MFormer network compresses and saves the current memory information K13 and V13 after processing the second intermediate video feature. The first layer of MFormer network outputs the third intermediate video feature after processing the second video segment in combination with K11 and V11 and compresses and saves the current memory information K21 and V21. The second layer of MFormer network outputs the fourth intermediate video feature after processing the third intermediate video feature in combination with K12 and V12 and compresses and saves the current memory information K22 and V22. The third layer of MFormer network compresses and saves the current memory information K23 and V23 after processing the fourth intermediate video feature in combination with K13 and V13. The above steps are sequentially performed on the third video segment and the fourth video segment. Finally, the video frame feature is output after the fourth video segment is processed by the three layers of MFormer network. The MFormer network mainly includes multi-head self-attention mechanism (Multi-head Self-Attention, MSA) and multilayer perceptron (Multilayer Perceptron, MLP). The specific process is as shown in Figure 3As shown, the fourth video segment is processed by the MFormer network in the figure. The input fourth video segment obtains feature functions V(t), K(t) and Q(t) through layer normalization Norm; V(t) and K(t) are processed through compression F(C) to obtain memory information K4 and V4 (which can be K41 and V41, K42 and V42, or K43 and V43), K4 and V4 are spliced with the memory information K3 and V3 (corresponding to K31 and V31, K32 and V32, or K33 and V33) of the third video segment into features K' and V', and then K' and V' are processed through attention Attention with Q(t). The formula of Attention is:

[0074]

[0075] where Softmax is a flexible maximum transmission function, Q is the query in the attention mechanism, c represents the channel dimension, K t is the memory information obtained by splicing the memory information of the current video segment and the memory information of the previous video segment, i.e., the key in the attention mechanism, and V t is the memory information obtained by splicing the memory information of the current video segment and the memory information of the previous video segment, i.e., the value in the attention mechanism. The calculation formulas of K t and V t are as follows:

[0076] K m , V m = Compress(K, V), K t = Concat(K m , K), V t = Concat(V m , V)

[0077] where K and V represent the memory information saved by the MFormer network when processing the current video segment, Compress represents one-dimensional convolution compression, K m and V m represent the compressed memory information, Concat is a splicing function, and K t and V t are the memory information obtained by splicing the memory information of the previous video segment and the memory information of the current video segment. In this embodiment, Compress is used to reduce the current memory information as much as possible to reduce the subsequent calculation amount.

[0078] Then, MSA is realized by the following formula:

[0079] MSA = concat(head1...head h )W 0, head i = Attention(Q; K t ; V t )

[0080] wherein head h denotes that there are h heads in the MSA, Concat is a concatenation function, W 0 is a weight matrix; head i represents a specific operation of one of the heads in head h , Q is a query in the attention mechanism, K t is a key in the attention mechanism, and V t is a value in the attention mechanism.

[0081] Finally, the MFormer network processing is realized by the following formula:

[0082] Y rgb = X 0 + MSA [LN(X 0 )], X 1 = Y rgb + MLP [LN(Y rgb )]

[0083] wherein Y rgb is an intermediate value, LN is layer normalization, X 0 is an input value, X 1 is an output value, MSA is a multi-head self-attention mechanism, and MLP is a multi-layer perception.

[0084] Since the MFormer network combines the memory information of the previous video segment when processing each video segment, the memory information combined by the last video segment includes the memory information of all previous video segments. For example, in this embodiment, the memory information K3 and V3 of the third video segment combined with the memory information K4 and V4 of the fourth video segment contains the memory information of the first video segment and the second video segment.

[0085] Step S14: performing feature fusion on the event segment features and the video frame features to obtain a recognition result.

[0086] In an embodiment, the step includes: concatenating a plurality of the event segment features and the video frame features into event stream features and RGB video features, respectively; and concatenating and fusing the event stream features and the RGB video features according to a channel dimension to obtain the recognition result.

[0087] Specifically, after obtaining a plurality of event segment features and video frame features, the event stream features and the RGB video features are spliced respectively; the event stream features and the RGB video features are spliced along the channel dimension, and then processed through two fully connected layers (FC) to obtain a recognition result Results after fusion of the dual modal data, as shown in the following formula: Figure 2

[0088] Step S15: adjusting the model parameters according to the loss function between the recognition result and the label to obtain a trained event recognition model.

[0089] Specifically, the recognition result and the label are iteratively trained by minimizing the loss function, and the model parameters are adjusted according to the loss value; after the loss value between the recognition result and the label is lower than a certain threshold, a trained event recognition model is obtained.

[0090] Figure 4 is a flowchart of an event recognition method provided by an embodiment of the application.

[0091] The application also provides an event recognition method, which combines Figure 4 The method comprises the following steps:

[0092] Step S41: obtaining event data to be recognized.

[0093] Step S42: inputting the event data into an event recognition model obtained by the training method of the event recognition model to obtain an event recognition result.

[0094] The application obtains event stream features and RGB video features by processing the event stream and the RGB video of the event sample data through the pulse convolutional neural network and the memory Transformer network of the event recognition model to be trained, and obtains a recognition result by fusing the dual modal data, and finally adjusts the model parameters in combination with the label of the event sample data, so that the finally trained event recognition model not only has time and space information, but also is accurate and high-performance.

[0095] It should be noted that the step division of the above methods is only for the purpose of clear description, and can be combined into one step or split into multiple steps in implementation, as long as the same logical relationship is included, and all are within the protection scope of the patent; adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the patent.

[0096] Figure 5 is a flowchart of a training system of an event recognition model provided by an embodiment of the application. ​

[0097] As Figure 5 shown, the device comprises an acquisition module 51, an event segment feature acquisition module 52, a video frame feature acquisition module 53, a result acquisition module 54 and a model adjustment module 55;

[0098] The acquisition module 51 is configured to acquire an event segment, a video frame and a label of event sample data;

[0099] The event segment feature acquisition module 52 is configured to input the event segment into a pulse convolutional neural network of an event recognition model to be trained to obtain event segment features;

[0100] The video frame feature acquisition module 53 is configured to input the video frame into a memory Transformer network of the event recognition model to be trained to obtain video frame features;

[0101] The result acquisition module 54 is configured to perform feature fusion on the event segment features and the video frame features to obtain an identification result;

[0102] The model adjustment module 55 is configured to adjust model parameters according to a loss function between the identification result and the label to obtain a trained event recognition model.

[0103] It should be noted that the event recognition model training system of the present embodiment is a system corresponding to the event recognition model training method described above, and the functional modules in the event recognition model training system correspond to the respective steps in the event recognition model training method. The event recognition model training system of the present embodiment can be implemented in cooperation with the event recognition model training method. Accordingly, the related technical details mentioned in the event recognition model training system of the present embodiment can also be applied in the event recognition model training method described above.

[0104] It should be noted that the above-mentioned functional modules can be integrated into one physical entity in whole or in part, or physically separated. These modules can be implemented in the form of software invoked by a processing element in whole, or in the form of hardware in whole, or in the form of software invoked by a processing element for part of the modules and in the form of hardware for part of the modules. In addition, these modules can be integrated together or implemented independently. The processing element mentioned herein can be an integrated circuit with signal processing capability. In the implementation process, part or all of the steps of the above method, or the above functional modules can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0105] The embodiments of the present application further provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the training method of the event recognition model provided in each of the above embodiments.

[0106] As shown in Figure 6 , the present application is a structural schematic diagram of the electronic device of the preferred embodiment used for implementing the training method of the event recognition model. It should be noted that, Figure 6 The computer system 600 of the electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0107] As shown in Figure 6 , the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage portion 608 to a random access memory (RAM) 603, such as performing the methods described in the above embodiments. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0108] The following components are connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 608 including a hard disk, and the like; and a communication portion 609 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 610 as needed, so that a computer program read therefrom is installed in the storage portion 608 as needed.

[0109] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with the embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising computer programs for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0110] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which the computer readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium that can send, propagate or transfer the program for use by or in connection with the instruction execution system, apparatus or device. The computer program contained on the computer readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, optical, or any suitable combination of the above.

[0111] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0112] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may

[0113] Another aspect of the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor of a computer, causes the computer to perform the method for training an event recognition model as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0114] Another aspect of the present application provides a computer program product or computer program, which comprises computer instructions. The computer instructions are stored in a computer readable storage medium. A processor of a computer reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method for training an event recognition model provided in the above embodiments.

[0115] It is apparent for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application.

[0116] The above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application, and although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application.

Claims

1. A training method for an event recognition model, characterized in that, include: Acquire event segments, video frames, and tags from event sample data; The event segment is input into the pulse convolutional neural network of the event recognition model to be trained to obtain the event segment features; The video frames are input into the memory Transformer network of the event recognition model to be trained to obtain video frame features; The event segment features and the video frame features are fused to obtain the recognition result; The model parameters are adjusted based on the loss function between the recognition result and the label to obtain the trained event recognition model; The event segment is input into the pulse convolutional neural network of the event recognition model to be trained to obtain event segment features, including: The event segment is input into the spiking convolutional neural network. The event segment is processed eight times by a neural network consisting of spiking neurons and two-dimensional convolutional layers. The first feature, the second feature, and the third feature are output after the eighth, sixth, and fourth processing, respectively. The first feature is processed through two layers of residual network and one layer of deconvolution layer and then concatenated with the second feature to obtain the first intermediate feature; The first intermediate feature is processed by a deconvolution layer and then concatenated with the third feature to obtain the second intermediate feature; The second intermediate feature is processed by a deformable convolutional neural network to obtain the event segment feature.

2. The training method for the event recognition model according to claim 1, characterized in that, The steps for obtaining event segments, video frames, and tags for event sample data include: Obtain the event sample data and the tags; The event stream and RGB video of the event sample data are processed to obtain the event segment and the video frame.

3. The training method for the event recognition model according to claim 1, characterized in that, The step of inputting the video frame into the memory Transformer network of the event recognition model to be trained to obtain the video frame features includes: The video frames are divided into multiple sequentially arranged video segments according to time. Each video segment and the memory information corresponding to the previous video segment are sequentially input into the memory Transformer network to obtain the corresponding intermediate video features and the memory information. The intermediate video features output by the memory Transformer network for the last video segment are used as the video frame features.

4. The training method for the event recognition model according to claim 1, characterized in that, The step of fusing the event segment features and the video frame features to obtain the recognition result includes: The event segment features and the video frame features are concatenated to form event stream features and RGB video features, respectively. The recognition result is obtained by concatenating and fusing the event stream features and the RGB video features according to the channel dimension.

5. An event recognition method, characterized in that, include: Obtain the event data to be identified; The event data is input into the event recognition model obtained by the training method of the event recognition model according to any one of claims 1 to 4, and the event recognition result is obtained.

6. A training system for an event recognition model, characterized in that, include: The acquisition module is used to acquire event segments, video frames, and tags from event sample data; The event segment feature acquisition module is used to input the event segment into the pulse convolutional neural network of the event recognition model to be trained, and obtain the event segment features; The video frame feature acquisition module is used to input the video frame into the memory Transformer network of the event recognition model to be trained, and obtain the video frame features. The result acquisition module is used to fuse the event segment features and the video frame features to obtain the recognition result. The model adjustment module is used to adjust the model parameters according to the loss function between the recognition result and the label to obtain the trained event recognition model; The event segment is input into the pulse convolutional neural network of the event recognition model to be trained to obtain event segment features, including: The event segment is input into the spiking convolutional neural network. The event segment is processed eight times by a neural network consisting of spiking neurons and two-dimensional convolutional layers. The first feature, the second feature, and the third feature are output after the eighth, sixth, and fourth processing, respectively. The first feature is processed through two layers of residual network and one layer of deconvolution layer and then concatenated with the second feature to obtain the first intermediate feature; The first intermediate feature is processed by a deconvolution layer and then concatenated with the third feature to obtain the second intermediate feature; The second intermediate feature is processed by a deformable convolutional neural network to obtain the event segment feature.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the training method of the event recognition model as described in any one of claims 1 to 4.

Images