An event identification method, system, device, and medium

Abstract

This invention relates to the field of event camera technology, specifically to an event recognition method comprising: acquiring an event stream to be recognized; preprocessing to obtain event frames corresponding to the event stream to be recognized, and preprocessing to obtain graph data corresponding to the event stream to be recognized; constructing an initial model of a bimodal fusion network; extracting the spatiotemporal features of the event frames and the global representation of the graph data, and concatenating and fusing the spatiotemporal features and the global representation so that the initial model of the bimodal fusion network learns the unified features of the bimodality to obtain a trained bimodal fusion network model; and using the bimodal fusion network model for event recognition. This invention, by constructing a bimodal fusion network, can better mine effective original information from the bimodality, learn the unified features of the bimodal data, effectively preserve the original information by converting the event stream into event frames and graph data, and improve the performance of event recognition by concatenating and fusing spatiotemporal features and global representations.

Description

Technical Field

[0001] This invention belongs to the field of event camera technology, specifically relating to an event recognition method, system, device, and medium. Background Technology

[0002] In recent years, event cameras have attracted increasing attention from researchers as a type of biosensor. Unlike RGB cameras, which record scenes synchronously into video frames, event cameras asynchronously trigger events by saving event points when the intensity change of each pixel exceeds a given threshold. Due to this unique imaging principle, event cameras exhibit advantages such as high dynamic range, low power consumption, and high temporal resolution. However, event cameras asynchronously generate events and noise, making it difficult for existing deep learning frameworks to directly process this unstructured data. Therefore, finding a method that can effectively process event data while preserving its original information as much as possible is particularly important.

[0003] Some mainstream algorithms choose to stack the data into 2D frames. While this allows for application to mainstream frameworks like CNNs after preprocessing, it loses the temporally dense features of the original event stream and lacks preservation of the original low-level information. Another transformation approach is to treat the event data as a graph or point cloud, maintaining the sparsity of the temporal data. However, the large scale of event data may introduce too much redundant information, and most algorithms that achieve classification through graph construction often use global pooling to obtain the final global representation. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the present invention constructs a bimodal fusion network, which can better mine the effective original information of bimodal data, learn the unified features of bimodal data, effectively preserve the original information by converting event streams into event frames and graph data, and improve the performance of event recognition by splicing and fusing spatiotemporal features and global representations without introducing too much redundant information.

[0005] To achieve the above and other related objectives, the present invention provides an event recognition method comprising: acquiring an event stream to be recognized; preprocessing the event stream to be recognized by stacking events in a temporal dimension to obtain event frames corresponding to the event stream to be recognized, and preprocessing the event stream to be recognized by constructing a voxel grid and plotting the graph to obtain graph data corresponding to the event stream to be recognized; constructing an initial model of a bimodal fusion network, the initial model of the bimodal fusion network including an event frame feature extraction model and a graph feature extraction model; extracting the spatiotemporal features of the event frames and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and concatenating and fusing the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of the bimodality to obtain a trained bimodal fusion network model; and using the bimodal fusion network model for event recognition.

[0006] In an optional embodiment of the present invention, the step of preprocessing the event stream to be identified by stacking events in the temporal dimension to obtain the event frames corresponding to the event stream to be identified, and preprocessing the event stream to be identified by constructing a voxel grid and plotting to obtain the graph data corresponding to the event stream to be identified includes: downsampling and plotting the event stream to be identified by constructing a voxel grid to obtain the graph data corresponding to the event stream to be identified.

[0007] In an optional embodiment of the present invention, the step of extracting the spatiotemporal features of the event frame and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and then concatenating and fusing the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of the bimodality and obtain the trained bimodal fusion network model includes: dividing each event frame into equal patch blocks according to spatial size; concatenating the patch blocks by frame to obtain the discrete token of the event frame; and inputting the discrete token into the event frame feature extraction model to extract the spatiotemporal features of the event frame.

[0008] In an optional embodiment of the present invention, in the step of inputting the discrete token into the ST-Transformer module and extracting the spatiotemporal features of the event frame using the event frame feature extraction model, the spatiotemporal features are extracted by the following formula:

[0009] Y out =X 0 +MSA(LN(X 0 ))

[0010] X 1 =Y out +MLP(LN(Y out ))

[0011] Among them, X 0 X represents the input of the current ST-Transformer. 1 This represents the output of the current ST-Transformer. LN(·) indicates layer normalization, MLP indicates multilayer perceptron, and MSA indicates multi-head self-attention mechanism.

[0012] In an optional embodiment of the present invention, the step of extracting the spatiotemporal features of the event frame and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and concatenating and fusing the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of the bimodality and obtain the trained bimodal fusion network model further includes: aggregating the features of each node corresponding to the graph data based on spatial graph convolution to obtain the high-level semantic representation of the graph data; and inputting the high-level semantic representation of the graph data into the graph feature extraction model to extract the global representation of the graph data.

[0013] In an optional embodiment of the present invention, in the step of aggregating the features of each node corresponding to the graph data based on spatial graph convolution to obtain a high-level semantic representation of the graph data, the feature aggregation formula is as follows:

[0014]

[0015] Where σ(.) represents the activation function, and f(.) represents the feature vector. Let v represent the learnable kernel function. The set of neighboring nodes.

[0016] In an optional embodiment of the present invention, the step of extracting the spatiotemporal features of the event frame and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and concatenating and fusing the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of the bimodality and obtain the trained bimodal fusion network model further includes: concatenating and fusing the spatiotemporal features and the global representation to obtain the unified features of the bimodality; and the initial model of the bimodal fusion network learning the unified features to obtain the trained bimodal fusion network model.

[0017] To achieve the above and other related objectives, the present invention also provides an event recognition system comprising: an acquisition module for acquiring an event stream to be recognized as raw data; a preprocessing module for preprocessing the event stream to be recognized by stacking events in a temporal dimension to obtain event frames corresponding to the event stream, and for preprocessing the event stream to be recognized by constructing a voxel grid and constructing a graph to obtain graph data corresponding to the event stream; a construction module for constructing an initial model of a bimodal fusion network, the initial model of the bimodal fusion network including an event frame feature extraction model and a graph feature extraction model; a fusion module for extracting spatiotemporal features of the event frames and global representations of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and concatenating and fusing the spatiotemporal features and global representations so that the initial model of the bimodal fusion network learns the unified features of the bimodality to obtain a trained bimodal fusion network model; and a recognition module for performing event recognition using the bimodal fusion network model.

[0018] The technical advantage of this invention lies in providing an event recognition method. By constructing a dual-modal fusion network, this invention can better mine effective information from dual modalities and learn the unified features of dual-modal data. By converting event streams into event frames and graph data, effective information is effectively preserved. Furthermore, the ST-Transformer module is used to process event frames, which can better represent the spatiotemporal features of the video. The graph neural network is used to process graph data, which can better represent the global representation of the video. Finally, the spatiotemporal features and global representation are spliced ​​and fused to improve the performance of event recognition. Attached Figure Description

[0019] Figure 1 This is an application scenario diagram of the event recognition method proposed in this invention;

[0020] Figure 2 Here is a flowchart of an event recognition method proposed in this invention;

[0021] Figure 3 This is a flowchart illustrating the specific process for extracting the spatiotemporal features of event frames as proposed in this invention.

[0022] Figure 4 This is a flowchart illustrating the global representation of graph data extraction proposed in this invention;

[0023] Figure 5 This is a flowchart illustrating the trained dual-modal fusion network model proposed in this invention.

[0024] Figure 6 Here is a flowchart of an event recognition method proposed in this invention;

[0025] Figure 7a This is a schematic diagram of the dual-modal fusion network model framework proposed in this invention;

[0026] Figure 7b This is a flowchart illustrating the graph feature extraction module proposed in this invention.

[0027] Figure 7c This is a flowchart illustrating the event frame feature extraction module proposed in this invention.

[0028] Figure 8 This is a functional block diagram of an event recognition system proposed in this invention;

[0029] Figure 9 This is a structural block diagram of the electronic device proposed in this invention. Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0033] Figure 1 The embodiment of the present invention provides an application scenario diagram of the event recognition method. The present invention preprocesses the event stream to be recognized to obtain event frames and graph data. The event frames utilize an event frame feature extraction module to obtain the spatiotemporal features of the event stream, and the graph data utilizes a graph data feature extraction module to obtain a global representation of the event stream. The spatiotemporal features and the global representation are then concatenated and fused to obtain a unified representation. In other application scenarios, the event recognition and classification can be set according to the actual situation, and the embodiments of the present invention do not impose any limitations on this.

[0034] The electronic device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0035] The electronic device may also include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0036] The networks in which the electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0037] Figure 2 This is a flowchart illustrating an event recognition method provided by an embodiment of the present invention. It should be noted that this invention is based on event recognition using Transformer and graph convolutional neural networks. This method can be applied to... Figure 1 The implementation environment shown is intended to illustrate a method that can also be applied to other instance implementation environments and executed by devices in those environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0038] like Figure 2 As shown, an event recognition method in this embodiment includes at least:

[0039] Step S21: Obtain the event stream to be identified. It should be noted that the event stream to be identified includes the video to be identified.

[0040] Step S22: Preprocess the event stream to be identified by stacking events in the temporal dimension to obtain the event frames corresponding to the event stream to be identified, and preprocess the event stream to be identified by constructing a voxel mesh and plotting the graph to obtain the graph data corresponding to the event stream to be identified. It should be noted that the event frames corresponding to the event stream to be identified are obtained by stacking events in the temporal dimension to preprocess the event stream to be identified.

[0041] In a specific embodiment, the initial representation of the event stream is (T, X, Y, P). Where T represents time, (X, Y) represents the coordinates in space, and P represents the polarity of the event. Record the coordinate points containing events in the 2D space. If the polarity of the event corresponding to this coordinate point is 1, set the RGB value of this point to the RGB value corresponding to red. If it is -1, set it to the RGB value corresponding to blue. The positions without events are recorded as 0. Divide along the event T dimension according to t. Then, in the time series dimension, convert the event segment with length t into a video. The video contains several event frames, and the original event stream is converted into videos.

[0042] In a specific embodiment, downsample and compose the to-be-recognized event stream by constructing a voxel grid to obtain the graph data corresponding to the to-be-recognized event stream.

[0043] Specifically, the initial representation of the event stream is (T, X, Y, P). Take (T, X, Y) as the three-dimensional coordinates of the event stream. T is in milliseconds. Normalize T to the magnitude of the spatial coordinates (X, Y). Through t i =A×(t i -t1) / (t n -t1), where n is the total number of events in the event stream and i is the event sequence number. The 3D coordinate range of the event stream is (A, X, Y). It is agreed that the size of each voxel is (l, h, w). Therefore, the 3D coordinate range of the constructed voxel set is (A / l, X / h, Y / w). Each voxel contains several original events. Concatenate the polarities of the events inside the voxel as the initial feature of the voxel. Because the number of original events contained in each voxel is inconsistent, we select k voxels with the largest number of original events from the generated voxel set for retention, and regard the other voxels as noise and discard them, so as to obtain a more robust graph structure representation.

[0044] Furthermore, regard each voxel as a node for graph composition. If the 3D coordinate distance between nodes is less than the preset threshold R, that is, d(v i , v j ) < R, then connect them with an edge. The calculation of d(v i , v j ) is as follows:

[0045]

[0046] Where, i represents the node, j represents the node, x, y, t are the spatial coordinates and time series coordinates corresponding to the node, d is a feature vector from 1 to D, and D is the dimension of the feature.

[0047] Furthermore, on the above basis, add an absorbing node. The added absorbing node is connected with edges to all other absorbing nodes.

[0048] Step S23: Construct the initial model of the bimodal fusion network. The initial model of the bimodal fusion network includes an event frame feature extraction model and a graph feature extraction model. It should be noted that the event frame feature extraction model includes multiple discrete tokens and multiple ST-Transformer modules, and the graph feature extraction module includes a graph convolution module, which includes multiple graph convolutional layers.

[0049] Step S24: Extract the spatiotemporal features of the event frames and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively. Then, concatenate and fuse the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of both modalities, thereby obtaining the trained bimodal fusion network model. It should be noted that three graph convolutional modules and three ST-Transformer modules respectively learn the features of the event frames and the graph data representation.

[0050] Step S25: Use a dual-modal fusion network model for event recognition.

[0051] like Figure 3 As shown, the specific steps for extracting the spatiotemporal features of event frames provided in this embodiment include:

[0052] Step S31: Divide each event frame into equal patch blocks according to spatial size. It should be noted that during the video recognition process, after data augmentation, each video yields 8 event frames of size 224×224. These event frames are then input into the residual pre-training model to extract initial features.

[0053] Step S32: Concatenate the patch blocks frame by frame to obtain the discrete token of the event frame. It should be noted that during the video recognition process, each event frame is divided into four equally sized patch blocks according to spatial size, and then concatenated frame by frame to obtain the discrete token representation of the video.

[0054] Step S33: Input the discrete token into the event frame feature extraction model to extract the spatiotemporal features of the event frame. It should be noted that the ST-Transformer module in the event frame feature extraction model includes a multi-head attention mechanism and an MLP to fully extract the spatiotemporal features of the event frame, specifically implemented through the following formula:

[0055] Y out =X 0 +MSA(LN(X 0 ))

[0056] X 1 =Yout +MLP(LN(Y out ))

[0057] Among them, X 0 X represents the input of the current ST-Transformer. 1 This represents the output of the current ST-Transformer. LN(·) indicates layer normalization, MLP indicates multilayer perceptron, and MSA indicates multi-head self-attention mechanism.

[0058] Furthermore, the expression for the multi-head self-attention mechanism is as follows:

[0059] MSE = concat(head1...head) h W 0

[0060] head i =Attention(Q; K; V)

[0061] Where Q, K, and V represent the output of LN(X0) obtained through three different linear mappings, h represents the number of heads in the multi-head self-attention mechanism, i represents the node, and W... 0 This represents the learnable parameter matrix.

[0062] like Figure 4 As shown, the specific steps for extracting the global representation of graph data provided in this embodiment include:

[0063] Step S41: Based on spatial graph convolution, feature aggregation is performed on the features of each node corresponding to the graph data to obtain a high-level semantic representation of the graph data. It should be noted that during the event recognition process, the graph data is input into a graph convolutional neural network to aggregate the features of each node in the graph. Since the number of events in each event stream is inconsistent, a Gaussian mixture model of spatial graph convolution is used as the core convolution kernel, expressed as follows:

[0064]

[0065] Where σ(.) represents the activation function, and f(.) represents the feature vector. Let v represent the learnable kernel function. The set of neighboring nodes, d is a feature vector from 1 to D, where D is the dimension of the feature.

[0066] Specifically, a graph convolution module includes three graph convolution layers. The first two layers realize the dimensionality increase of features to obtain a high-level semantic representation of graph data, and the third layer is used for residual connection to prevent overfitting and gradient vanishing.

[0067] Step S42: Input the high-level semantic representation of the graph data into the graph feature extraction model to extract the global representation of the graph data.

[0068] like Figure 5 As shown, the specific steps for obtaining the trained bimodal fusion network model provided in this embodiment include:

[0069] Step S51: The spatiotemporal features and global representations are concatenated and fused to obtain the unified features of the bimodality. It should be noted that the three graph convolutional modules and the three ST-Transformer modules respectively learn the features represented by event frames and voxel maps. The absorption node features output by each graph convolutional module (i.e., the global representation) are concatenated with the token sequence (i.e., the spatiotemporal representation) input by each ST-Transformer. Information fusion is then performed through the ST-Transformer module to obtain the unified features of the bimodality, thereby realizing bimodal learning and interaction.

[0070] Step S52: The initial model of the bimodal fusion network learns the unified feature to obtain the trained bimodal fusion network model. It should be noted that the output token sequence of the last ST-Transformer module is average-pooled and then processed through two fully connected layers to obtain the trained bimodal fusion network model.

[0071] like Figure 6 As shown in Figure 7, the technical solution of the present invention will be described below with reference to specific embodiments:

[0072] Figure 6 This is a flowchart illustrating the specific process of an event recognition method proposed in this invention. The process begins with inputting a stream of events to be recognized. The stream is preprocessed by stacking events along a temporal dimension and constructing a voxel grid and graph, resulting in event frame representations and graph data. The event frames are then input into a residual training model. The ST-Transformer module processes the event frames, extracting the spatiotemporal features of the event stream (spatiotemporal representation in the diagram). Virtual nodes are added to the graph data. The graph data with virtual nodes is input into a graph convolutional neural network. The graph convolutional neural network performs feature aggregation on the graph data (voxel graph in the diagram), using the absorbed node features as the global representation of the graph. The spatiotemporal and global representations are concatenated and fused, enabling the initial model of the bimodal fusion network to learn a unified bimodal representation. This unified representation is input into a negative log-likelihood loss function. The initial model of the bimodal fusion network is iteratively trained to minimize the loss function to obtain the bimodal fusion network model. The bimodal fusion network model is then used to recognize the event stream, outputting the recognition structure, and the event recognition process ends.

[0073] Figure 7aThis is a schematic diagram of the dual-modal fusion network model framework proposed in this invention. First, the event stream to be identified is preprocessed to obtain event frames and graph data of the event stream to be identified. Then, an initial model of the dual-modal fusion network is constructed. The event stream obtains spatiotemporal features through the event frame feature extraction module, and the graph data obtains global representation through the graph feature extraction module. The spatiotemporal features and global representation are concatenated and fused to obtain a unified representation. The initial model of the dual-modal fusion network learns the unified representation to obtain the dual-modal fusion network model. Event recognition is then performed using the dual-modal fusion network model. Figure 7b This is a flowchart of the graph feature extraction module, which is mainly used to extract global representations for subsequent use as input to the ST-Transformer module. Figure 7c This is a flowchart of the event frame feature extraction module, which is mainly used to extract spatiotemporal representations for subsequent use as input to the next ST-Transformer module.

[0074] like Figure 8 As shown, the present invention also provides an event recognition system comprising: an acquisition module 81, a preprocessing module 82, a construction module 83, a fusion module 84, and a recognition module 85. The acquisition module 81 acquires the event stream to be recognized; the preprocessing module 82 preprocesses the event stream to be recognized by stacking events in the temporal dimension to obtain event frames corresponding to the event stream, and preprocesses the event stream to be recognized by constructing a voxel grid and graphing to obtain graph data corresponding to the event stream; the construction module 83 constructs an initial model of a bimodal fusion network, which includes an event frame feature extraction model and a graph feature extraction model; the fusion module 84 uses the event frame feature extraction model and the graph feature extraction model to extract the spatiotemporal features of the event frames and the global representation of the graph data, respectively, and splices and fuses the spatiotemporal features and the global representation so that the initial model of the bimodal fusion network learns the unified features of the bimodality to obtain the trained bimodal fusion network model; the recognition module 85 uses the bimodal fusion network model for event recognition.

[0075] It should be noted that the embodiments provided above are as follows: Figure 8 The event recognition system shown here belongs to the same concept as the event recognition method provided in the above embodiments. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the event recognition system provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0076] Embodiments of the present invention also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the event recognition method provided in the above embodiments.

[0077] Figure 9 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown. It should be noted that... Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0078] like Figure 9 As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 902 or programs loaded from storage portion 906 into Random Access Memory (RAM) 903, such as performing the methods described in the above embodiments. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0079] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 906 including a hard disk, etc.; and a communication section 908 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 908 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 906 as needed.

[0080] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 908, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of the present invention.

[0081] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0084] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the event recognition method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0085] Another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the event recognition methods provided in the various embodiments described above.

[0086] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An event recognition method, characterized in that, include: Acquire the event stream to be identified, the event stream including the video to be identified; The event frames corresponding to the event stream to be identified are obtained by stacking events in the temporal dimension, and the graph data corresponding to the event stream to be identified is obtained by constructing a voxel mesh and constructing a graph. An initial model for a dual-modal fusion network is constructed, which includes an event frame feature extraction model and a graph feature extraction model. The event frame feature extraction model and the graph feature extraction model are used to extract the spatiotemporal features of the event frame and the global representation of the graph data, respectively. The spatiotemporal features and the global representation are then concatenated and fused so that the initial model of the bimodal fusion network learns the unified features of the bimodality to obtain the trained bimodal fusion network model. Event recognition is performed using a dual-modal fusion network model.

2. The event recognition method according to claim 1, characterized in that, The steps of preprocessing the event stream to be identified by stacking events in the temporal dimension to obtain event frames corresponding to the event stream to be identified, and preprocessing the event stream to be identified by constructing a voxel mesh and drawing a graph to obtain graph data corresponding to the event stream to be identified, include: The event stream to be identified is downsampled and plotted by constructing a voxel mesh to obtain the graph data corresponding to the event stream to be identified.

3. The event recognition method according to claim 1, characterized in that, The steps of extracting spatiotemporal features of event frames and global representations of graph data using the event frame feature extraction model and the graph feature extraction model respectively, and then concatenating and fusing the spatiotemporal features and global representations to enable the initial model of the bimodal fusion network to learn the unified features of the bimodalities, thereby obtaining the trained bimodal fusion network model, include: Each event frame is divided into equal patch blocks according to its spatial size; The patch blocks are concatenated frame by frame to obtain the discrete tokens of the event frames; The discrete token is input into the event frame feature extraction model to extract the spatiotemporal features of the event frame.

4. The event recognition method according to claim 3, characterized in that, In the step of inputting the discrete token into the ST-Transformer module and extracting the spatiotemporal features of the event frame using the event frame feature extraction model, the spatiotemporal features are extracted using the following formula: in, Represents the input of the current ST-Transformer. This represents the output of the current ST-Transformer, and LN(·) denotes layer normalization. This represents a multilayer perceptron. This represents a multi-head self-attention mechanism.

5. The event recognition method according to claim 1, characterized in that, The step of extracting the spatiotemporal features of the event frame and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and then concatenating and fusing the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of the bimodalities, thereby obtaining the trained bimodal fusion network model, further includes: Based on spatial domain graph convolution, the features of each node corresponding to the graph data are aggregated to obtain a high-level semantic representation of the graph data. The high-level semantic representation of the graph data is input into the graph feature extraction model to extract the global representation of the graph data.

6. The event recognition method according to claim 5, characterized in that, In the step of aggregating the features of each node corresponding to the graph data based on spatial domain graph convolution to obtain a high-level semantic representation of the graph data, the feature aggregation formula is as follows: ( ) in, This represents the activation function. Represents the eigenvector. This represents a learnable kernel function. express The set of neighboring nodes, d represents a feature vector from 1 to D, and D represents the dimension of the feature.

7. The event recognition method according to claim 1, characterized in that, The step of extracting the spatiotemporal features of the event frame and the global representation of the graph data using the event frame feature extraction model and the graph feature extraction model respectively, and then concatenating and fusing the spatiotemporal features and the global representation to enable the initial model of the bimodal fusion network to learn the unified features of the bimodalities, thereby obtaining the trained bimodal fusion network model, further includes: The spatiotemporal features and global representations are spliced ​​and fused to obtain the unified features of the dual modes; The initial model of the bimodal fusion network learns the unified features to obtain the trained bimodal fusion network model.

8. An event recognition method, characterized in that, include: The acquisition module acquires the event stream to be identified, which includes the video to be identified; The preprocessing module preprocesses the event stream to be identified by stacking events in the temporal dimension to obtain the event frames corresponding to the event stream to be identified, and preprocesses the event stream to be identified by constructing a voxel mesh and drawing a graph to obtain the graph data corresponding to the event stream to be identified; The module constructs an initial model for the dual-modal fusion network, which includes an event frame feature extraction model and a graph feature extraction model. The fusion module uses the event frame feature extraction model and the graph feature extraction model to extract the spatiotemporal features of the event frame and the global representation of the graph data, respectively. It then concatenates and fuses the spatiotemporal features and the global representation so that the initial model of the bimodal fusion network learns the unified features of the bimodality to obtain the trained bimodal fusion network model. The recognition module uses a dual-modal fusion network model for event recognition.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the event recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the event recognition method according to any one of claims 1 to 7.

Images