A method and system for recognizing events in a video

Abstract

The application discloses a method and system for identifying events in a video. The method comprises: obtaining static features of each frame in the video based on a neural network; obtaining spatial interaction relationship features corresponding to each static feature based on a spatial attention mechanism network; obtaining space-time interaction relationship features corresponding to the set of spatial interaction relationship features based on a time domain attention mechanism network; constructing a space-time interaction relationship graph based on the space-time interaction relationship features, and obtaining features of the space-time interaction relationship graph through a graph convolutional neural network; and identifying events in the video based on the space-time interaction relationship features and the features of the space-time interaction relationship graph. The scheme of the application improves the accuracy and efficiency of identifying events in the video.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for identifying events in a video, a computer device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computers, image acquisition devices, and wireless transmission technology, video has become the most common form of streaming media in our daily lives. Video surveillance is a comprehensive system with strong preventative capabilities and a crucial component of security systems. Its intuitive, accurate, timely, and informative nature has led to its widespread application in many applications. For example, surveillance cameras in cities can capture video footage from various scenes. However, while these cameras can capture vast amounts of video, they lack the corresponding supervisory and auditing capabilities, often requiring manual screening. Therefore, as video surveillance security systems become more widespread, they increasingly require the use of computer vision methods to intelligently identify, analyze, and understand video content and the actual active objects within them (intelligent video analysis). This allows for preliminary screening of these massive amounts of video, reducing the workload of supervisors and auditors and improving the accuracy and efficiency of video audits.

[0003] With the continuous advancement of deep learning, the development of internet technology, and the widespread adoption of smart devices, intelligent video analysis has become a research hotspot in the field of computer vision. Detecting abnormal events in intelligent video analysis—detecting events in video sequences caused by monitored targets, such as people or cars, that differ significantly from the expected event model—has become a core task in intelligent surveillance systems.

[0004] Generally speaking, intelligent video analysis involves tasks such as action recognition and instance segmentation, but identifying unusual events in videos is the most challenging of these tasks. This is because: the duration of an event is uncertain, with significant variations in duration distribution across different events; the scenes and environments involved are varied and complex; and the objects involved are diverse. Consequently, many models that excel in action or behavior recognition tasks fail to achieve ideal results for event recognition.

[0005] Therefore, how to accurately and efficiently identify events in videos has become one of the problems that need to be solved urgently. Summary of the Invention

[0006] The present invention provides a method, system, computer device and computer-readable storage medium for identifying events in a video, so as to accurately and efficiently identify events in a video in different scenarios.

[0007] The present invention provides a method for identifying events in a video, comprising:

[0008] Obtain static features of each frame in the video based on neural network;

[0009] Based on the spatial attention mechanism network, the spatial interaction relationship features corresponding to each static feature are obtained;

[0010] Based on the time domain attention mechanism network, the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set are obtained;

[0011] Based on the spatiotemporal interaction relationship features, a spatiotemporal interaction relationship graph is constructed, and the features of the spatiotemporal interaction relationship graph are obtained through a graph convolutional neural network;

[0012] Recognize events in videos based on spatiotemporal interaction relationship features and spatiotemporal interaction relationship graph features.

[0013] Optionally, the method for identifying events in a video further includes:

[0014] The static features of each frame are divided into blocks and spatial positions are embedded;

[0015] The spatial attention mechanism network-based method for obtaining spatial interaction relationship features corresponding to each static feature includes: using a frame of static features that are category-labeled, block-parted, and spatially embedded as input to the spatial attention mechanism network to obtain spatial interaction relationship features corresponding to the static features of the frame.

[0016] Optionally, the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set obtained based on the temporal attention mechanism network include:

[0017] Perform temporal position embedding on the spatial interaction feature set;

[0018] The spatial interaction feature set with category labels and temporal position embedding is used as the input of the temporal attention mechanism network to obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction feature set.

[0019] Optionally, constructing a spatiotemporal interaction relationship graph based on spatiotemporal interaction relationship features includes:

[0020] Performing linear transformation on the spatiotemporal interaction relationship features to obtain the features of the nodes of the spatiotemporal interaction relationship graph;

[0021] The adjacency matrix of the spatiotemporal interaction graph is obtained based on the inverse of the distance between node features after linear transformation.

[0022] Optionally, the identifying of events in the video based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph includes:

[0023] After performing residual operations on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph, events in the video are identified based on the multi-layer perceptron layer and the classification layer.

[0024] Optionally, each frame in the video is a key frame in the video, and the method for identifying an event in the video further includes:

[0025] Sampling the captured video at a preset frame number to obtain a first frame sequence;

[0026] extracting a key frame from the first frame sequence based on a clustering method and discarding similar frames in the same cluster that are similar to the key frame to obtain a second frame sequence;

[0027] A plurality of continuous frames are selected from the second frame sequence to obtain a key frame sequence.

[0028] The present invention also provides a system for identifying events in a video, comprising:

[0029] Neural network, used to obtain static features of each frame in the video;

[0030] Spatial attention mechanism network, used to obtain the spatial interaction relationship features corresponding to each static feature;

[0031] The temporal attention mechanism network is used to obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set;

[0032] A construction unit, used for constructing a spatiotemporal interaction relationship graph based on spatiotemporal interaction relationship features;

[0033] Graph convolutional neural network, used to obtain the features of spatiotemporal interaction graph;

[0034] The recognition unit is used to recognize events in the video based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph.

[0035] The present invention also provides a computer device comprising at least one processor and at least one memory, wherein the memory stores a computer program, which, when executed by the processor, enables the processor to perform the above-mentioned method for identifying events in a video.

[0036] The present invention also provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor in a device, the device is enabled to perform the above-mentioned method for identifying events in a video.

[0037] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0038] The method for identifying events in a video described above in the present invention first obtains static features of each frame in the video based on a neural network. Then, spatial interaction relationship features corresponding to each static feature are obtained based on a spatial attention mechanism network. Spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set are obtained based on a temporal attention mechanism network. A spatiotemporal interaction relationship graph is constructed based on the spatiotemporal interaction relationship features, and features of the spatiotemporal interaction relationship graph are obtained through a graph convolutional neural network. Finally, events in the video are identified based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph. Because the static features of each frame in the video are first obtained through a neural network and the structural information of the image is extracted, the loss of local information in the spatial domain of each frame in the video is avoided. After obtaining the static features of each frame in the video, an adaptive attention mechanism network (spatial attention mechanism network and temporal attention mechanism network) is used to extract the spatiotemporal interaction relationship features of each frame in the video. This allows for adaptive display representation and fusion of objects, environments, behaviors, concepts, and interactions contained in the video, thereby obtaining the most representative spatiotemporal interaction relationship features in the video frames. Then, when a spatiotemporal interaction relationship graph is constructed based on the spatiotemporal interaction relationship features, the spatiotemporal interaction relationship graph can explicitly represent the mid-level semantics of the video. Finally, when the features of the spatiotemporal interaction relationship graph are extracted through a graph convolutional neural network, the interactive relationship of the mid-level semantics of the video can be further extracted, and the logical reasoning of semantic information at a deeper level can be realized. When events in the video are subsequently identified based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph, classification errors caused by unclear semantic features can be avoided, thereby improving the accuracy of identifying events in the video. In addition, since there is no need to manually verify the events in the video, the efficiency of identifying events in the video is also greatly improved.

[0039] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 Schematic diagram of a flow chart of a method for identifying events in a video according to an embodiment of the present invention;

[0043] Figure 2A schematic diagram of a process for obtaining spatial interaction relationship features corresponding to static features according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of the temporal attention mechanism network according to an embodiment of the present invention;

[0045] Figure 4 Schematic diagram of the process of identifying events in a video according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0047] As described in the prior art, the current event recognition involved in intelligent video analysis, such as identifying potential or ongoing illegal or criminal events, and reviewing videos according to requirements, etc. The existing recognition models are lacking in recognition accuracy and recognition speed, and the recognition results are not ideal. Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for identifying events in a video according to an embodiment of the present invention. Figure 1 As shown, the method for identifying events in a video includes:

[0048] S101: Obtain static features of each frame in the video based on a neural network.

[0049] S102: Obtain the spatial interaction relationship features corresponding to each static feature based on the spatial attention mechanism network.

[0050] S103: Obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set based on the temporal attention mechanism network.

[0051] S104: Construct a spatiotemporal interaction relationship graph based on the spatiotemporal interaction relationship features, and obtain the features of the spatiotemporal interaction relationship graph through a graph convolutional neural network.

[0052] S105: Identify events in the video based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph.

[0053] Execute S101 to obtain static features of each frame in the video through a neural network. The static features can be underlying features such as shape, color, texture, etc. in each frame. Specifically, in this embodiment, a deep convolutional neural network (DCNN) such as ResNet18, ResNet34, and ResNet50 can be used to perform multi-layer convolution operations on the input video frame image to extract the static features X of each video frame. i , i=1,2,...,N.

[0054] In this embodiment, in order to improve the speed and accuracy of identifying events in the video, only events in the key frames in the captured video can be identified. The key frames are frames that contain valid information in the video. For different scenes and different needs, the identification of valid information is also different. In actual application, it can be determined according to actual needs. In this embodiment, the key frames in the captured video can be obtained by the following methods:

[0055] First, the captured video is sampled at a preset frame number to obtain a first frame sequence. Specifically, the preset frame number F count Determine the sampling rate. In this embodiment, the preset number of frames F sampled in 1000 frames is set. count ∈[65,74], then the sampling rate The collected video is sampled at a sampling rate f to obtain a sampled frame sequence F sample That is, the first frame sequence. In this embodiment, in order to reduce the amount of calculation and facilitate the subsequent processing of the video frames by the neural network, the height and width of each frame image in the first frame sequence can be scaled to [256, 320].

[0056] Then, based on the clustering method, key frames are extracted from the first frame sequence and similar frames in the same cluster that are similar to the key frame are discarded to obtain the second frame sequence. Specifically, the initial threshold k = 0.9 can be set (k is a parameter measuring the similarity between two adjacent frames) to obtain the second frame sequence from the first frame sequence F. sample After extracting the key frame, discard the similar frames in the same cluster that are similar to the key frame to obtain the second frame sequence. count ∈[65,74], therefore, the number of key frames in the second frame sequence extracted Should meet Therefore, in the process of extracting key frames (obtaining the second sequence), if the number of key frames If the above conditions are not met, the threshold value k needs to be continuously adjusted, and the first frame sequence F is re-set according to the adjusted new threshold value k. sampleExtract key frames, discard similar frames to obtain the second frame sequence, until the number of key frames in the second frame sequence meets In this embodiment, in order to discard similar frames in the same cluster that are similar to the key frame, 0.2≤k≤1.0 can be set. In this embodiment, when k=1, it means that every frame in the video is retained. If the number of key frames is If f*65 is not reached, the number of key frames will be interpolated to reach f*65. When k=0.2, it means that similar frames have been discarded to the maximum extent during cluster sampling, and the frames retained are all representative frames. If the number of key frames is If it exceeds f*74, some frames will be discarded according to actual needs so that the number of key frames reaches f*74.

[0057] Finally, multiple consecutive frames are selected from the second frame sequence. In this embodiment, 64 consecutive frames can be selected as the key frame sequence. In order to reduce the amount of calculation and facilitate the subsequent processing of the video frame by the neural network, in this embodiment, each frame in the key frame sequence can be cropped. Specifically, for each frame, a random cropping starting point can be determined from the shorter side, and the size of each frame can be cropped to [224, 224]. The cropped key frame sequence F = [F1, F2, F3, ..., F N ] as the final key frame sequence input to the neural network.

[0058] Execute S102, and obtain the spatial interaction relationship features corresponding to each static feature based on the spatial attention mechanism network. In one embodiment, specifically, after obtaining the static feature X of the key frame in the video through S101, i After that, we can first analyze the static features X of each key frame. i Divide into blocks, and then perform block X i Perform spatial position embedding to obtain the input sequence of the input spatial attention mechanism network Where i = 1, 2, 3, ..., N, and n is the number of blocks in the feature map after division. In this embodiment, in order to reduce the amount of calculation and improve the processing efficiency in the process of obtaining the spatial interaction relationship features corresponding to each static feature, for each key frame that has been divided and spatially embedded, a CLS category tag is added to the initial position of the spatial position embedding to represent the semantic information extracted by the spatial attention mechanism network, and then the CLS and As the input of the spatial attention mechanism network, the spatial interaction relationship features corresponding to the static features of each key frame are obtained through the spatial attention mechanism network. The spatial interaction relationship features corresponding to the static features of all key frames constitute the spatial interaction relationship feature set

[0059] See also Figure 2 , Figure 2 Schematic diagram of the process of obtaining spatial interaction relationship features corresponding to static features according to an embodiment of the present invention. Figure 2 As shown, the static features X of the video frame (key frame) extracted by the deep convolutional neural network i After block segmentation and spatial position embedding, the input sequence is obtained Right now For each key frame, the CLS category marker is added to the initial position of the spatial position embedding ( Figure 2 The box on the right of each frame before inputting the spatial Transformer is the CLS category marker. Figure 2 Each frame after block and position embedding is also the input of the spatial Transformer. Figure 2 (not shown) and use the two as input to the spatial Transformer to obtain the spatial interaction relationship features corresponding to the static features of each key frame The spatial interaction features are averaged and pooled and then input into the time domain Transformer.

[0060] Execute S103, and obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set based on the temporal attention mechanism network. In one embodiment, in order to facilitate the temporal attention mechanism network to process the spatial interaction feature set, the spatial interaction relationship features output by the spatial attention mechanism network are first averaged and pooled to reduce Then the spatial interaction relationship feature set Perform time domain position embedding to obtain Add the CLS category marker to the initial position of the time domain position embedding, and then combine CLS and X ST As the input of the time domain attention mechanism network, the spatiotemporal interaction relationship feature R corresponding to the spatial interaction relationship feature set is obtained through the time domain attention mechanism network. ST In this embodiment, the structure of the time domain attention mechanism network can be Pre-LN Transformer. Figure 3 , Figure 3 Schematic diagram of the structure of the time domain attention mechanism network of an embodiment of the present invention. Figure 3In the figure, the left figure is a schematic diagram of the structure of the time-domain attention mechanism network, the middle figure is a schematic diagram of the structure of the multi-head attention module, and the right figure is a schematic diagram of the structure of the scaled dot product attention module. In this embodiment, the time-domain attention mechanism network may include a normalization layer, L attention modules and a feedforward neural network. Specifically, the attention module may be a multi-head attention module. The input of the first-layer attention module is the normalized input feature, and the output of the first-layer attention module and the input feature are residually connected as the output of the first layer. The output of the first layer is normalized and used as the input of the second-layer attention module, and the output of the second-layer attention module and the output of the first layer are residually connected as the output of the second layer, and so on, until the output of the Lth layer is obtained. The output of the Lth layer is normalized and input into the feedforward neural network, and the output of the feedforward neural network and the output of the Lth layer are residually connected as the output of the time-domain attention mechanism network.

[0061] Continue to see Figure 3 In the middle figure, the multi-head attention module includes multiple parallel scaled dot product attention modules. The structural diagram of each scaled dot product attention module is as follows Figure 3 As shown in the right figure, the input of the scaled dot product attention module is the Query, Key and Value matrices (the Query element is used to split the query, the Key element is used to calculate the global matching degree, and the Value is used to represent the features of each dimension). Generally speaking, when inputting for the first time, you can set Q = K = V, and divide Q, K, and V on the last dimension according to the number of heads n set to obtain the split:

[0062] Q=[q1 q2...q i ...q n ] T ,K=[k1 k2...k i ...k n ] T ,V=[v1 v2...v i ...v n ] T , for every q i , and K T Perform matrix dot multiplication to calculate q i The degree of matching with the global features is scaled and Softmaxed, and then multiplied with V to obtain the Attention() function: Similarly, Q and K T Perform matrix dot multiplication, scale and Softmax, and then dot multiplication with V to obtain the corresponding Attention() function: In this embodiment, the time domain Transformer adopts the Pre-LN Transformer structure, so before entering the attention module, the normalization layer will first be based on the dimension d of K k The input Q, K, V is processed, so the attention function of the scaled dot product attention module is finally:

[0063] From the above and Figure 3 It can be seen that the multi-head attention module includes multiple parallel scaled dot product attention modules. Figure 3 , the scaled dot product attention module performs parameter mapping on the Q, K, V matrices, and then calculates the h scaled dot product attention modules separately, and concatenates and linearly transforms the calculation results to obtain the output of the multi-head attention module. In the multi-head attention module, the Attention() function of each head is the final Attention() function of the scaled dot product attention module mentioned above, which can be expressed as: head i =Attention(QW i Q ,KW i K ,VW i V ),in, is the mapping parameter matrix of the input features, so the output of a multi-head attention module can be expressed as: Multihead=Concat(head1,head2,...,head h ), so after the feature matrix passes through a multi-head attention module, a feature matrix with the same dimension as the original input matrix will be obtained. In this embodiment, The spatiotemporal interaction relationship feature RST∈R is obtained after inputting into the time domain Transformer (L multi-head attention modules) B×N′×C , where B is the input batch size, N′ is the number of frames N of the input segment plus the position embedding of the temporal CLS marker, so N′ = N + 1, and C is the output feature dimension. In this embodiment, the spatial attention mechanism network can also adopt the Pre-LN Transformer structure, which will not be repeated here.

[0064] Execute S104, after obtaining the spatiotemporal interaction relationship feature R through S103 ST Then, based on the spatiotemporal interaction feature R ST Constructing the spatiotemporal interaction graph G ST Then, the spatiotemporal interaction relationship graph G is obtained through the graph convolutional neural network ST In this embodiment, the spatiotemporal interaction relationship diagram G ST It can be expressed as G ST=(V,A), where node V = Value = φ V (R ST ),φ V is a linear transformation, node V∈R N′×C The adjacency matrix A is obtained based on the inverse of the distance between the node features after linear transformation, such as the Euclidean distance between the node features after linear transformation. In addition, in this embodiment, since the interaction relationship between nodes in the constructed spatiotemporal interaction relationship graph is bidirectional, the constructed spatiotemporal interaction relationship graph is an undirected graph. Therefore, the adjacency matrix can be specifically: A ij =A ji , A ii =0

[0065]

[0066] Among them, v i , v j is the node feature, f A To perform linear transformation on node features, W A ∈R C×C is the fully connected parameter matrix.

[0067] After determining the spatiotemporal interaction graph G ST After expressing it, it is input into the graph convolutional neural network to obtain the characteristics of the spatiotemporal interaction relationship graph

[0068] In this embodiment, the graph convolution layer of the graph convolutional neural network can be expressed as H l+1 =σ(L sym H l W l ), where σ(·) is the activation function, H l is the lth activation matrix th , H l ∈R N′×C , H 1 =V R , W l is a matrix trained for a specific layer. In this embodiment, the Laplace matrix I n is the feature matrix, D ii =∑ j A ij Is the degree matrix, then the above graph convolution layer can be expressed as: The characteristics of the spatiotemporal interaction relationship graph finally output by the graph convolutional neural network

[0069] Execute S105 to obtain the features of the spatiotemporal interaction relationship graph through the graph convolutional neural network Then, based on the spatiotemporal interaction feature R ST and the characteristics of the spatiotemporal interaction graph Identify events in the video. Specifically, R ST and After performing residual connections, the multi-layer perception layer (MLP) and classification layers, such as the Softmax layer, are used to predict the category of events in the video, and then the events in the video can be identified, such as whether the event is an abnormal event.

[0070] So far, the above S101 to S105 have realized the recognition of events in the video. Figure 4 The process of identifying events in a video according to an embodiment of the present invention is briefly described. Figure 4 FIG. 1 is a schematic diagram of a process for identifying events in a video according to an embodiment of the present invention. Figure 4 As shown, the key frames of the collected video clips are extracted to obtain the key frame sequence F1, F2, ..., F N , and then extract the static features X1, X2, ..., X of each key frame in the key frame sequence through the neural network ResNet50 i ,…,X N , for each key frame static feature X i Perform block segmentation and spatial position embedding to obtain For each key frame that has been segmented and spatially embedded, a CLS category tag is added to the initial position of the spatial embedding, and then They are used as inputs of each spatial Transformer to obtain the spatial interaction relationship features corresponding to the static features of each key frame. The spatial interaction feature set After performing temporal position embedding (not shown in the figure) and adding the CLS category tag to the initial position of the temporal position embedding, it is input into the temporal Transformer to obtain the spatiotemporal interaction relationship feature R corresponding to the spatial interaction relationship feature set. ST According to the spatial-temporal interaction characteristics R ST To construct the spatiotemporal interaction graph G ST , using graph convolutional neural network to obtain spatiotemporal interaction relationship graph G ST Features Finally, the spatiotemporal interaction feature R ST and the characteristics of the spatiotemporal interaction graph Perform residual connection, and then use MLP layer and Softmax layer to realize the recognition of video clips.

[0071] In another embodiment, the present invention also provides a system for identifying events in videos, the system comprising: a neural network, a spatial attention mechanism network, a temporal attention mechanism network, a construction unit, a graph convolutional neural network, and an identification unit, wherein: the neural network is used to obtain the static features of each frame in the video. The spatial attention mechanism network is used to obtain the spatial interaction relationship features corresponding to each static feature. The temporal attention mechanism network is used to obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set. The construction unit is used to construct a spatiotemporal interaction relationship graph based on the spatiotemporal interaction relationship features. The graph convolutional neural network is used to obtain the features of the spatiotemporal interaction relationship graph. The identification unit is used to identify events in the video based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph. The implementation of the system for identifying events in videos of this embodiment can refer to the implementation of the above-mentioned method for identifying events in videos, which will not be repeated here.

[0072] Based on the same technical concept, an embodiment of the present invention provides a computer device comprising at least one processor and at least one memory, wherein the memory stores a computer program, which, when executed by the processor, enables the processor to execute the above-mentioned method for identifying events in a video.

[0073] Based on the same technical concept, an embodiment of the present invention provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor in a device, the device can perform the above-mentioned method for identifying events in a video.

[0074] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0076] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0078] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0079] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for identifying events in a video, characterized in that include: Obtain static features of each frame in the video based on neural network; Based on the spatial attention mechanism network, the spatial interaction relationship features corresponding to each static feature are obtained; Based on the time domain attention mechanism network, the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set are obtained; Based on the spatiotemporal interaction relationship features, a spatiotemporal interaction relationship graph is constructed, and the features of the spatiotemporal interaction relationship graph are obtained through a graph convolutional neural network; Recognize events in videos based on spatiotemporal interaction relationship features and spatiotemporal interaction relationship graph features; Also includes: The static features of each frame are divided into blocks and spatial positions are embedded; The method of obtaining spatial interaction relationship features corresponding to each static feature based on the spatial attention mechanism network includes: using a frame of static features that are category-labeled, block-parted, and spatially embedded as input to the spatial attention mechanism network to obtain spatial interaction relationship features corresponding to the static features of the frame; The spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set obtained based on the temporal attention mechanism network include: Perform temporal position embedding on the spatial interaction feature set; The spatial interaction feature set with category labels and temporal position embedding is used as the input of the temporal attention mechanism network to obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction feature set; Constructing a spatiotemporal interaction relationship graph based on spatiotemporal interaction relationship features includes: Performing linear transformation on the spatiotemporal interaction relationship features to obtain the features of the nodes of the spatiotemporal interaction relationship graph; The adjacency matrix of the spatiotemporal interaction graph is obtained based on the inverse of the distance between node features after linear transformation.

2. The method according to claim 1, wherein The feature recognition of events in the video based on the spatiotemporal interaction relationship features and the spatiotemporal interaction relationship graph includes: After performing residual operations on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph, events in the video are identified based on the multi-layer perceptron layer and the classification layer.

3. The method according to claim 1, wherein Each frame in the video is a key frame in the video, and the method further includes: Sampling the captured video at a preset frame number to obtain a first frame sequence; extracting a key frame from the first frame sequence based on a clustering method and discarding similar frames in the same cluster that are similar to the key frame to obtain a second frame sequence; A plurality of continuous frames are selected from the second frame sequence to obtain a key frame sequence.

4. The method according to claim 1, wherein The temporal attention mechanism network includes a normalization layer, L attention modules and a feedforward neural network; The attention module is a multi-head attention module, wherein the input of the first layer attention module is the normalized input feature, the output of the first layer attention module and the input feature are residually connected as the output of the first layer, the output of the first layer is normalized and used as the input of the second layer attention module, the output of the second layer attention module and the output of the first layer are residually connected as the output of the second layer, and so on, until the output of the Lth layer is obtained; The output of the Lth layer is normalized and input into the feedforward neural network. The output of the feedforward neural network and the output of the Lth layer are residually connected to form the output of the time-domain attention mechanism network.

5. The method according to claim 4, wherein The multi-head attention module includes multiple parallel scaled dot product attention modules. The input of each scaled dot product attention module is three matrices: Query, Key and Value. The Query element is used to split the query, the Key element is used to calculate the global matching degree, and the Value is used to characterize the features of each dimension. When inputting for the first time, set Q = K = V, and divide Q, K, and V along the last dimension according to the number of heads n set, to obtain the following: Q=[q1 q2 ...q i ... q n ] T ,K=[k1 k2 ...k i ... k n ] T ,V=[v1 v2 ...v i ... v n ] T ; For each q i , and K T Perform matrix dot multiplication to calculate q i The degree of matching with the global features is scaled and Softmaxed, and then multiplied with V to obtain the Attention() function: Similarly, Q and K T Perform matrix dot multiplication, scale and Softmax, and then dot multiplication with V to obtain the corresponding Attention() function: The temporal transformer uses the Pre-LN transformer structure. Before inputting the attention module, the normalization layer firstly transforms the input data according to the dimension d of K. k After processing the input Q, K, V, the attention function of the scaled dot product attention module is finally:

6. A system for identifying events in a video, characterized in that: include: Neural network, used to obtain static features of each frame in the video; Spatial attention mechanism network, used to obtain the spatial interaction relationship features corresponding to each static feature; The temporal attention mechanism network is used to obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction relationship feature set; A construction unit, used for constructing a spatiotemporal interaction relationship graph based on spatiotemporal interaction relationship features; Graph convolutional neural network, used to obtain the features of spatiotemporal interaction graph; an identification unit, configured to identify events in a video based on the spatiotemporal interaction relationship features and the features of the spatiotemporal interaction relationship graph; The neural network further includes: dividing the static features of each frame into blocks and embedding the spatial positions; The spatial attention mechanism network includes: taking a frame of static features that are category-labeled, block-parted, and spatially embedded as input to the spatial attention mechanism network to obtain spatial interaction relationship features corresponding to the frame of static features; The temporal attention mechanism network includes: Perform temporal position embedding on the spatial interaction feature set; The spatial interaction feature set with category labels and temporal position embedding is used as the input of the temporal attention mechanism network to obtain the spatiotemporal interaction relationship features corresponding to the spatial interaction feature set; The building block comprises: The spatiotemporal interaction relationship features are linearly transformed to obtain the features of the nodes of the spatiotemporal interaction relationship graph; and the adjacency matrix of the spatiotemporal interaction relationship graph is obtained based on the inverse of the distance between the node features after the linear transformation.

7. A computer device comprising at least one processor and at least one memory, wherein: The memory stores a computer program, and when the program is executed by the processor, the processor is enabled to perform the method for identifying events in a video according to any one of claims 1 to 5.

8. A computer-readable storage medium, when instructions in the storage medium are executed by a processor in a device, enables the device to perform the method for identifying events in a video according to any one of claims 1 to 5.

Images