A group behavior recognition method based on complementary spatiotemporal information modeling

Abstract

The present invention discloses a group behavior recognition method based on complementary spatiotemporal information modeling. The method comprises: obtaining individual feature vectors in a target video; inputting the individual feature vectors into a trained group behavior recognition model to obtain a group behavior recognition result, wherein the group behavior recognition model comprises a first modeling branch and a second modeling branch, the first modeling branch sequentially obtains enhanced individual features through a first spatial self-attention module and a first temporal self-attention module for the input individual features, and then recognizes all enhanced individual features to obtain a first group behavior recognition result; the second modeling branch sequentially obtains enhanced individual features through a second temporal self-attention module and a second spatial self-attention module for the input individual features, and then recognizes all enhanced individual features to obtain a second group behavior recognition result. The present invention improves the accuracy of group behavior recognition and enhances the robustness of the model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically, to a group behavior recognition method based on complementary spatiotemporal information modeling. Background Art

[0002] The concept of deep learning originates from the study of artificial neural networks. For example, a multi-layer perceptron with multiple hidden layers is a deep learning structure. Deep learning combines low-level features to form more abstract high-level representations of attribute categories or features to discover distributed feature representations of data. Deep machine learning methods are divided into supervised learning and unsupervised learning. Different learning models are established under different learning frameworks. For example, a convolutional neural network (CNN) is a machine learning model under deep supervised learning.

[0003] Group behavior recognition is an important research problem that is widely used in the field of computer vision and needs to be solved urgently. With the development of deep learning technology, the breadth and depth of group behavior recognition and understanding are also expanding. Group behavior recognition refers to the technology of judging and classifying the behavior state of the target group in the video by analyzing the individual behavior content of the objects contained in the video. In the analysis and understanding of the video, spatiotemporal information modeling refers to the information interaction of the pixel points of each frame of the video or the feature points obtained by deep learning within and between frames.

[0004] In the prior art, the paper (DOI: 10.1109 / CVPR42600.2020.00092, Actor-Transformers for Group Activity Recognition) proposed to use the Transformer model to model the similarity of different target objects in the video and enhance their features, thereby improving the accuracy of group behavior recognition. The basic idea of ​​this paper is: 1) Use a deep neural network to extract deep features from different frames of the video, and then use the target bounding box to extract the deep features of all targets in different frames; 2) The deep features of all targets obtained are subjected to embedded operations and position encoding, and input into the autocorrelation attention module to obtain enhanced target individual features; 3) Use classifiers to distinguish different targets separately to obtain individual behavior categories; fuse the features of different targets to obtain group deep features, and use different classifiers to distinguish and obtain group behavior categories. By testing on international general video group behavior recognition datasets (VolleyballDataset, Collective Dataset and NBAdataset), the accuracy of group behavior recognition in videos has been improved. However, existing video group behavior recognition only considers one order of spatiotemporal modeling, ignoring that spatiotemporal modeling can be achieved in two orders and that these two modeling methods are highly complementary. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a group behavior recognition method based on complementary spatiotemporal information modeling. The method comprises the following steps:

[0006] Obtain individual feature vectors in the target video;

[0007] The individual feature vector is input into a trained group behavior recognition model to obtain a group behavior recognition result, wherein the group behavior recognition model includes a first modeling branch and a second modeling branch, the first modeling branch obtains enhanced individual features through a first spatial self-attention module and a first temporal self-attention module for the input individual features, and then identifies all enhanced individual features to obtain a first group behavior recognition result; the second modeling branch obtains enhanced individual features through a second temporal self-attention module and a second spatial self-attention module for the input individual features, and then identifies all enhanced individual features to obtain a second group behavior recognition result; the group behavior recognition result is obtained by fusing the first group behavior result and the second group behavior recognition result.

[0008] Compared with the prior art, the advantage of the present invention is that it is the first time to use two complementary spatiotemporal relationship modeling to solve the problem of group behavior category recognition errors, thereby enhancing the robustness of the model. In addition, by designing contrast loss functions between frames, between frames and videos, and between videos, the consistency of feature expressions learned by the two modeling branches of time-space and space-time can be constrained, further improving the accuracy of group behavior recognition.

[0009] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0011] Figure 1 is a flow chart of a group behavior recognition method based on complementary spatiotemporal information modeling according to an embodiment of the present invention;

[0012] Figure 2 is a general architecture diagram of a group behavior recognition method based on complementary spatiotemporal information modeling according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of a multi-level contrast loss function according to an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of an application scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0016] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0017] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0018] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0019] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0020] See also Figure 1 As shown, the provided group behavior recognition method based on complementary spatiotemporal information modeling includes the following steps.

[0021] Step S110: constructing a group behavior recognition model, which includes two modeling branches with a dual spatiotemporal relationship.

[0022] Combination Figure 2 As shown, the constructed group behavior recognition model includes a space-time modeling branch and a time-space modeling branch. The two branches model the time and space relationship respectively by using the self-attention mechanism, and are stacked in two different orders, namely space-time and time-space, to construct a complementary space-time relationship with duality.

[0023] For the space-time branch (or ST branch), the input individual features are successively enhanced by the spatial attention module and the temporal attention module, and can be further divided into group features and individual behaviors, and then the group behavior recognition results are obtained based on the group features. For the time-space branch (or TS branch), the input individual features are successively enhanced by the temporal attention module and the spatial attention module, and can be divided into group features and individual behaviors, and then the group behavior recognition results are obtained based on the group features. The final group behavior recognition result can be obtained by fusing the behavior recognition results of the two modeling branches. The dual spatiotemporal modeling is realized through the two designed branches.

[0024] In one embodiment, the individual features of the two modeling branches are obtained by: extracting K video frames from the input video, each of which has N individuals, and obtaining the feature vector X of all individuals through a deep neural network and RoiAlign (region of interest alignment), that is, K×N×C dimensional features, where C represents the latitude of the deep feature vector. ROIAlign is a regional feature aggregation method used to determine the target bounding box, which will not be described in detail here.

[0025] The temporal self-attention module is used for temporal relationship modeling. For example, the specific temporal relationship modeling method is: for each individual's K frame features, that is, K×C features, K feature relationships are constructed through the self-attention mechanism, and the feature expression is enhanced by the feedforward neural network (FFN) to obtain the individual features enhanced by temporal relationship modeling.

[0026] The spatial self-attention module is used for spatial relationship modeling. For example, the specific spatial relationship modeling method is: for the N individual features in each video frame, that is, N×C features, the relationship between N features is constructed through the self-attention mechanism, and the feature expression is enhanced by the FFN network to obtain the individual features after spatial relationship modeling enhancement.

[0027] In the embodiment of the present invention, the specific process of dual space-time modeling includes:

[0028] Step S1: input the original individual features X into the spatial relationship modeling and the temporal relationship modeling in sequence to obtain all individual features X after the spatial-temporal relationship modeling. ST , input the classifier and obtain the group behavior recognition results of the space-time branch.

[0029] Step S2: input the original individual features X into the time relationship modeling and the space relationship modeling in sequence to obtain all individual features X after the time-space relationship modeling. TS , input the classifier and obtain the group behavior recognition results of the time-space branch.

[0030] Step S3, merging the recognition results of the two modeling branches to obtain the final group behavior recognition result.

[0031] It should be noted that in the embodiment of the present invention, the two modeling branches have a dual complementary relationship, the temporal self-attention modules on each branch can have the same or different structures, and the spatial self-attention modules can also have the same or slightly different structures. For example, the number of stacked self-attention modules can be different.

[0032] Step S120, training the group behavior recognition model based on the video sample set. During the training process, a multi-level contrast loss function is designed to constrain the consistency of feature expressions of the two modeling branches.

[0033] In order to further improve the accuracy of group behavior recognition, the consistency of feature expressions learned by the time-space and space-time modeling branches can be constrained during model training. Considering that the features of the same individual under different time-space modeling should be as consistent as possible, and at the same time different from the features of other individuals, multi-level feature constraints from local to global are designed for the two modeling branches.

[0034] For example, contrastive loss functions between frames, between frames and videos, and between videos are designed to constrain the consistency of feature expressions learned by the two modeling branches. Figure 3 is a schematic diagram of the contrast loss function, where Figure 3 (a) is the individual contrast loss function between frames, Figure 3 (b) is the individual contrast loss function between frames and videos, Figure 3 (c) is the individual contrast loss function between videos.

[0035] In one embodiment, the contrast loss function between frames is set to:

[0036]

[0037] Among them, h represents the cosine similarity CosSim, represents the features of the kth frame of the nth individual on the ST branch, represents the feature of the kth frame of the nth individual in the TS branch, and t represents the tth frame of the current video. For the nth individual in the ST branch, this function can maximize the consistency between the feature in its kth frame and its feature in the kth frame of the TS branch.

[0038] In one embodiment, the contrast loss function between frames and videos is:

[0039]

[0040] in, represents the video-level features of the nth individual in the TS branch, represents the video-level features of the i-th individual in all videos in the TS branch batch, B represents the batch size, that is, the number of videos contained in a batch of data, and N represents the number of individuals. The video-level features of the nth individual in the current video in the TS branch are obtained by max pooling the K frame features of the nth individual. For the nth individual in the ST branch, this function can maximize the consistency between its features in the kth frame and its video-level features in the TS branch.

[0041] In one embodiment, the contrast loss function between videos is set to:

[0042]

[0043] in, They represent the video-level features of the nth individual in the current video. This function can be used to maximize the consistency of the video features of the same individual in the ST and TS branches.

[0044] During the model training process, preferably, the three loss functions can be fused to obtain an overall loss function, with the goal of maximizing the distribution consistency of the two modeling branches between frames, between frames and videos, and between videos to obtain the optimal parameters of the model. For example, the overall loss function is designed as the direct addition or weighted fusion of the three loss functions.

[0045] It should be understood that, under the premise of designing dual spatiotemporal modeling, using a loss function or a fusion of two loss functions as the overall loss function can also improve the recognition accuracy of the model to a certain extent. In addition, during the model training process, different parameters can be set according to the training efficiency and model accuracy requirements, for example, K = 3, N = 12, C = 1024.

[0046] Step S130: Use the trained group behavior recognition model to perform behavior recognition on the target video.

[0047] The trained model can be used for group behavior recognition in actual target videos. In actual recognition, the multi-level contrast loss function is not calculated, and the rest is basically the same as the training process, which will not be repeated here.

[0048] The trained model obtained by the present invention can be applied to the client or server to realize group behavior recognition or analysis in target videos in various scenarios. Figure 4As shown. The server can be a computer, a cluster or a cloud server. The client can be an electronic device such as a smart phone, a tablet computer, a smart wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.), a smart car device, a computer, etc. The computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, etc.

[0049] To further verify the effect of the present invention, experiments were conducted. The experimental results show that on VolleyballDataset, Collective Dataset and NBA Dataset data, compared with the prior art, the present invention achieves the best recognition results. At the same time, the dual spatiotemporal relationship modeling module proposed in the present invention can be combined with any spatiotemporal modeling group behavior recognition method, and combined with feature constraints based on multi-level contrast loss functions, the group behavior recognition accuracy of the algorithm can be further improved and the dependence on the amount of training data can be reduced.

[0050] The present invention can be applied to a variety of scenarios, such as smart city security. The behavior patterns of people in the surveillance video are different, and the high-risk behaviors of the crowd, such as fighting, illegal assembly, etc., can be identified more effectively. At the same time, the present invention has a low demand for training data and is more suitable for identifying rare high-risk group behaviors. For example, in the scenario of urban intelligent security, the behavior of people within the monitoring range is accurately identified to assist in the identification and alarm of behaviors that endanger the city. In autonomous driving, at traffic intersections, the identification of the behavior patterns of people is very important for safe autonomous driving behavior. For another example, for monitoring wild animals, a large number of cameras are installed in some key animal protection and wildlife research areas. Group behavior recognition technology is an important basis for the automation of wildlife status detection.

[0051] In summary, compared with the prior art, the present invention has the following advantages:

[0052] 1) For the first time, two complementary dual spatiotemporal modeling methods are used to model the behavior of individuals in the video, which improves the accuracy of behavior recognition. However, the existing technology only models the spatiotemporal relationship through time-space or space-time, ignoring the complementarity of the two orders of spatiotemporal modeling.

[0053] 2) The present invention constrains the consistency of individual features between two modeling branches from three levels: between frames, between frames and videos, and between videos. That is, a multi-level contrast loss function is used. This design further improves the recognition accuracy and robustness of the model and reduces the dependence on training data. The traditional contrast loss function only focuses on the consistency between videos.

[0054] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0055] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0056] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0057] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0058] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0060] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0061] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0062] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A group behavior recognition method based on complementary spatiotemporal information modeling, comprising the following steps: Obtain individual feature vectors in the target video; The individual feature vector is input into a trained group behavior recognition model to obtain a group behavior recognition result, wherein the group behavior recognition model includes a first modeling branch and a second modeling branch, the first modeling branch and the second modeling branch have a dual complementary relationship, the first modeling branch obtains enhanced individual features through a first spatial self-attention module and a first temporal self-attention module for the input individual features, and then identifies all enhanced individual features to obtain a first group behavior recognition result; the second modeling branch obtains enhanced individual features through a second temporal self-attention module and a second spatial self-attention module for the input individual features, and then identifies all enhanced individual features to obtain a second group behavior recognition result; the group behavior recognition result is obtained by fusing the first group behavior result and the second group behavior recognition result; In the process of training the group behavior recognition model, the contrast loss function between frames, the contrast loss function between frames and videos, and the contrast loss function between videos are integrated to constrain the consistency of features between the first modeling branch and the second modeling branch from three levels; The contrast loss function between frames is set as: Among them, h represents the cosine similarity CosSim, represents the feature of the kth frame of the nth individual on the first modeling branch, represents the feature of the k-th frame of the n-th individual on the second modeling branch, t represents the index of the frame, and K represents the number of frames; The contrast loss function between the frame and the video is set as: in, represents the video-level features of the nth individual on the second modeling branch, represents the video-level features of the i-th individual in all videos in a batch on the second modeling branch, B represents the number of videos contained in a batch, N represents the number of individuals, Represents the features of the k-th frame of the n-th individual on the first modeling branch; The contrast loss function between the videos is set as: in, Represent the video-level features of the nth individual in the video on the second modeling branch and the first modeling branch respectively.

2. The method according to claim 1, characterized in that The individual feature vector is obtained by the following steps: For the target video, K video frames are extracted, each of which contains N individuals; The N individuals are aligned through a deep neural network and the region of interest RoiAlign to obtain the feature vectors of the N individuals.

3. The method according to claim 1, characterized in that The first spatial self-attention module and the second spatial self-attention module have the same or different structures, and the first temporal self-attention module and the second temporal self-attention module have the same or different structures.

4. The method according to claim 1, characterized in that: The first temporal self-attention module and the second temporal self-attention module are used for temporal relationship modeling. For each individual's multi-frame features, multiple feature relationships are constructed through a self-attention mechanism, and then input into a feedforward neural network to enhance feature expression, thereby obtaining individual features enhanced by temporal relationship modeling; The first spatial self-attention module and the second spatial self-attention module are used for spatial relationship modeling. For multiple individual features in each video frame, the relationship between the multiple individual features is constructed through the self-attention mechanism, and the feature expression is enhanced by inputting the feedforward neural network to obtain the individual features after spatial relationship modeling enhancement.

5. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

Images