Method and system for crowd behavior recognition of surveillance videos

Abstract

The application discloses a crowd behavior recognition method and system for monitoring video. The method comprises the following steps: inputting pedestrian crowd training video data set into a pedestrian action feature network to obtain action features and position features of each pedestrian in continuous time; inputting the action features into a time action evolution network to obtain action evolution features of each pedestrian in continuous time; inputting the action features and the position features into a spatial relationship network to obtain spatial relationship features of all pedestrians in a video image in a single time; fusing the spatial relationship features, the action features and the position features to obtain first fusion features, inputting the first fusion features into the time action evolution network to obtain first branch features; fusing the action evolution features, the action features and the position features to obtain second fusion features, inputting the second fusion features into the spatial relationship network to obtain second branch features, taking average to obtain double-branch features, inputting the double-branch features into a pedestrian crowd classification network to obtain pedestrian action classification and crowd behavior classification.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for recognizing group behavior in surveillance videos. Background Technology

[0002] In recent years, with the rapid pace of urbanization, the number of vehicles has gradually increased. However, due to a severe shortage of parking spaces, problems such as parking difficulties and traffic congestion have arisen. To standardize the management of roadside parking, high-position video technology has been developed. By installing high-position video cameras on the roadside, vehicle parking can be judged and managed, thus playing a positive role in promoting urban traffic management. Based on existing high-position video surveillance equipment, the obtained monitoring data, and current visual processing algorithms, it is not only possible to charge and manage roadside parking spaces, but also to identify vehicle violations, cloned license plates, and restricted vehicles. This allows for more effective use of high-position video technology to promote the healthy development of urban traffic. Furthermore, it can also identify the behavior of roadside pedestrians, promptly identifying illegal, dangerous, disturbing public order, and sudden falls, thereby contributing to intelligent traffic management and public security.

[0003] However, traditional methods rely solely on the time-dimensional features of individual pedestrians when performing human behavior recognition. This makes it difficult to accurately identify group behaviors among multiple pedestrians, and it is easy to miss key features, resulting in low recognition accuracy of traditional methods. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problem of low accuracy in traditional methods. To achieve the above objective, this invention provides a method and system for identifying group behavior in surveillance videos.

[0005] This invention provides a method for identifying group behavior in surveillance videos, comprising:

[0006] Individual and group behavior annotations were performed on roadside surveillance videos to obtain a training video dataset of pedestrian groups.

[0007] The pedestrian group training video dataset is input into the pedestrian action feature network to obtain the action features and position features of each pedestrian in a continuous time period.

[0008] The action features of each pedestrian within the continuous time period are input into the temporal action evolution network to obtain the action evolution features of each pedestrian within the continuous time period;

[0009] The action features and position features of all pedestrians in a video image within a single time period are input into a spatial relationship network to obtain the spatial relationship features of all pedestrians in the video image within the single time period;

[0010] The spatial relationship features, the action features, and the position features are fused to obtain a first fused feature, and the first fused feature is input into the temporal action evolution network to obtain a first branch feature;

[0011] The action evolution feature, the action feature, and the position feature are fused to obtain a second fused feature, and the second fused feature is input into the spatial relationship network to obtain a second branch feature;

[0012] The first branch feature and the second branch feature are averaged to obtain the bi-branch feature;

[0013] The dual-branch features are input into the pedestrian group classification network to obtain pedestrian action classification and group behavior classification;

[0014] A model loss function is constructed based on the pedestrian action classification, the group behavior classification, the first branch feature, and the second branch feature. The initial group behavior recognition model is trained to obtain a group behavior recognition model. The group behavior recognition model includes the pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network.

[0015] Based on the group behavior recognition model, group behavior recognition is performed on the roadside surveillance video under test to obtain the pedestrian action classification and the group behavior classification.

[0016] In one embodiment, inputting the pedestrian group training video dataset into a pedestrian action feature network to obtain the action features and location features of each pedestrian over a continuous time period includes:

[0017] The pedestrian group training video dataset is input into a convolutional neural network, which outputs the features of each video frame image within the continuous time period.

[0018] Based on the features of each video frame image within the continuous time period, pedestrian detection is performed using the target detection network to obtain the action features and position features of each pedestrian within the continuous time period.

[0019] In one embodiment, fusing the spatial relationship features, the action features, and the position features to obtain a first fused feature, and inputting the first fused feature into the temporal action evolution network to obtain a first branch feature, includes:

[0020] The first fused feature is obtained by summing the spatial relationship features, the action features, and the position features, or by splicing them together along the channel dimension.

[0021] In one embodiment, fusing the action evolution features, the action features, and the position features to obtain a second fused feature, and inputting the second fused feature into the spatial relationship network to obtain a second branch feature, includes:

[0022] The second fused feature is obtained by summing the action evolution feature, the action feature, and the position feature or by splicing them together in the channel dimension.

[0023] In one embodiment, the step of constructing a model loss function based on the pedestrian action classification, the group behavior classification, the first branch features, and the second branch features, and training the initial group behavior recognition model to obtain the group behavior recognition model includes:

[0024] Based on the pedestrian action classification and the group behavior classification, a cross-entropy classification loss function is constructed;

[0025] Based on the features of the first branch and the features of the second branch, a cosine similarity loss function is constructed;

[0026] The model loss function is formed based on the cross-entropy classification loss function and the cosine similarity loss function.

[0027] In one embodiment, the present invention provides a group behavior recognition system for surveillance video, comprising:

[0028] The pedestrian group data module is used to annotate individual and group behaviors in roadside surveillance videos to obtain a pedestrian group training video dataset.

[0029] The pedestrian action feature network module is used to input the pedestrian group training video dataset into the pedestrian action feature network to obtain the action features and position features of each pedestrian in a continuous time period.

[0030] The temporal action evolution network module is used to input the action features of each pedestrian within the continuous time period into the temporal action evolution network to obtain the action evolution features of each pedestrian within the continuous time period.

[0031] The spatial relationship network module is used to input the action features and position features of all pedestrians in the video image within a single time period into the spatial relationship network to obtain the spatial relationship features of all pedestrians in the video image within the single time period;

[0032] The first branch module is used to fuse the spatial relationship features, the action features, and the position features to obtain a first fused feature, and input the first fused feature into the time-action evolution network to obtain a first branch feature;

[0033] The second branch module is used to fuse the action evolution features, the action features, and the position features to obtain a second fused feature, and input the second fused feature into the spatial relationship network to obtain a second branch feature;

[0034] The dual-branch feature acquisition module is used to average the first branch feature and the second branch feature to obtain the dual-branch feature.

[0035] The pedestrian group classification network module is used to input the dual-branch features into the pedestrian group classification network to obtain pedestrian action classification and group behavior classification;

[0036] The model training module is used to construct a model loss function based on the pedestrian action classification, the group behavior classification, the first branch feature, and the second branch feature, and to train the initial group behavior recognition model to obtain a group behavior recognition model. The group behavior recognition model includes the pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network.

[0037] The behavior acquisition module is used to perform group behavior recognition on the roadside surveillance video under test according to the group behavior recognition model, and obtain the action classification of pedestrians and the behavior classification of the group.

[0038] In one embodiment, the pedestrian motion feature network module includes:

[0039] The image feature acquisition module is used to input the pedestrian group training video dataset into the convolutional neural network and output the features of each video frame image in the continuous time period;

[0040] The action location feature acquisition module is used to perform pedestrian detection based on the features of each video frame image within the continuous time period using the target detection network, and to obtain the action features and location features of each pedestrian within the continuous time period.

[0041] In one embodiment, the first branch module includes:

[0042] The first fusion feature acquisition module is used to sum or concatenate the spatial relationship features, the action features, and the position features in the channel dimension to obtain the first fusion feature.

[0043] In one embodiment, the second branch module includes:

[0044] The second fusion feature acquisition module is used to sum or concatenate the action evolution feature, the action feature, and the position feature in the channel dimension to obtain the second fusion feature.

[0045] In one embodiment, the model training module includes:

[0046] The first loss function module is used to construct a cross-entropy classification loss function based on the pedestrian action classification and the group behavior classification.

[0047] The second loss function module is used to construct a cosine similarity loss function based on the first branch features and the second branch features;

[0048] The model loss function module is used to form the model loss function based on the cross-entropy classification loss function and the cosine similarity loss function.

[0049] The aforementioned method and system for recognizing group behavior in surveillance videos obtains the action and location features of each pedestrian within a continuous time period through a pedestrian action feature network. These features are then further extracted using a temporal action evolution network and a spatial relationship network, yielding the action evolution features of each pedestrian within a continuous time period and the spatial relationship features of all pedestrians in a single video image within a single time period. Behavior represents the actions of a single person or multiple people over a relatively long period, while action represents the actions of a single person within a relatively short period. Furthermore, a dual-branch structure is constructed, first time-first, then space-first, and vice versa, to obtain dual-branch features. This enables spatial feature modeling for multiple pedestrians in each video frame, and also allows for feature modeling of each pedestrian across multiple video frames, resulting in a clearer understanding of the actions of multiple pedestrians. Finally, a pedestrian group classification network classifies the actions of individual pedestrians and groups based on the dual-branch features, leading to a more comprehensive and accurate understanding of pedestrian behavior in the video.

[0050] Therefore, the group behavior recognition method for surveillance videos provided by this invention enables the recognition of the behavior of each pedestrian within a continuous video sequence, thereby achieving the recognition and understanding of the behavior of multiple people over a period of time. When dangerous behavior or other sudden unsafe behavior is identified, timely warnings can be sent to relevant departments, allowing for the handling of the relevant behavior. This enables intelligent traffic management and public security management, and also avoids dangerous situations occurring late at night or in special road sections with low pedestrian traffic. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the steps of the method for identifying group behavior in surveillance video provided by the present invention.

[0052] Figure 2 This is a schematic diagram of the structure of the group behavior recognition system for surveillance video provided by the present invention. Detailed Implementation

[0053] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0054] Please see Figure 1 This invention provides a method for identifying group behavior in surveillance videos, comprising:

[0055] S10, label individual and group behaviors in roadside surveillance videos to obtain a pedestrian group training video dataset;

[0056] S20, input the pedestrian group training video dataset into the pedestrian action feature network to obtain the action features and position features of each pedestrian in a continuous time period;

[0057] S30, input the action features of each pedestrian in a continuous time period into the temporal action evolution network to obtain the action evolution features of each pedestrian in a continuous time period;

[0058] S40, input the motion features and position features of all pedestrians in the video image within a single time period into the spatial relationship network to obtain the spatial relationship features of all pedestrians in the video image within a single time period;

[0059] S50, the spatial relationship features, action features and position features are fused to obtain the first fused feature, and the first fused feature is input into the temporal action evolution network to obtain the first branch feature;

[0060] S60, the action evolution features, action features and position features are fused to obtain the second fused feature, and the second fused feature is input into the spatial relationship network to obtain the second branch feature;

[0061] S70, take the average of the first branch feature and the second branch feature to obtain the bi-branch feature;

[0062] S80, input the dual-branch features into the pedestrian group classification network to obtain pedestrian action classification and group behavior classification;

[0063] S90, construct a model loss function based on pedestrian action classification, group behavior classification, first branch features and second branch features, train the initial group behavior recognition model to obtain the group behavior recognition model, which includes a pedestrian action feature network, a temporal action evolution network, a spatial relationship network and a pedestrian group classification network.

[0064] S100 performs group behavior recognition on the roadside surveillance video under test based on the group behavior recognition model, and obtains the action classification of pedestrians and the behavior classification of the group.

[0065] In this embodiment, in S10, the pedestrian group training video dataset consists of multiple short video clips, each video clip including K frames. Two-dimensional object detection boxes and pedestrian action categories are labeled for each pedestrian in each image of each video clip, and group behavior categories are labeled for each video clip. The two-dimensional object detection box label for each pedestrian can be understood as describing the coordinates of the upper left and lower right corners of the pedestrian's two-dimensional rectangle. Pedestrian action categories can be understood as actions of a single pedestrian, such as raising an arm, kicking a leg, jumping, or falling. Group behavior categories can be understood as an action involving multiple people gathered together.

[0066] In S20, the pedestrian action feature network can be a convolutional neural network. The input to the pedestrian action feature network is each video frame image in the video segments of the pedestrian group training video dataset, and the output is the action feature representation and position feature representation of each pedestrian in a continuous time period.

[0067] In S30, the Temporal Action Evolution Network is used to model the action changes of each pedestrian across consecutive temporal image frames of a short video clip. The input to the Temporal Action Evolution Network is the action feature vector of a pedestrian across K consecutive frames, denoted as P, where P∈R. K*C Here, K represents K frames of images, C represents a motion feature vector with dimension C, and the output is the motion evolution features of a pedestrian in a series of video frames. Temporal motion evolution networks include, but are not limited to, learning motion evolution features using convolutional neural networks or Transformers.

[0068] In S40, the spatial relationship network is used to model the spatial relationships between all pedestrians in a single video image. The input to the spatial relationship network is the motion feature vector representation of all pedestrians in a given video frame, along with the positional feature vector of each pedestrian. The output is the spatial relationships between all pedestrians in the single video image. The spatial relationship network can be a graph convolutional neural network, treating each pedestrian in a single video image as a node in a graph structure, and the positional relationships between each pedestrian as edges between nodes, thus forming a spatial graph structure data. Leveraging the powerful learning ability of graph convolutional neural networks for graph structure data, it enables the prediction of the spatial relationships between multiple pedestrians in a video image within a single time frame.

[0069] In S50 and S60, a dual-branch structure is established. One branch is to fuse the spatial relationship features output by the spatial relationship network, the action features obtained by the pedestrian action feature network, and the position features as a whole to form the first fused feature. The first fused feature is then input into the temporal action evolution network to obtain the first branch feature.

[0070] Branch 2 involves fusing the action evolution features output by the temporal action evolution network, the action features obtained from the pedestrian action feature network, and the location features as a whole to form a second fused feature. This second fused feature is then input into the spatial relationship network to obtain the second branch feature.

[0071] In S70, the results output by the two-branch structure are averaged between the first branch feature and the second branch feature to obtain the two-branch feature.

[0072] In S80, the pedestrian group classification network consists of pedestrian action classification and group behavior classification parts, constructing pedestrian action classifiers and group behavior classifiers. The pedestrian action classifier and group behavior classifier in the pedestrian group classification network can each be constructed using fully connected neural networks, with inputs being bi-branch features and outputs being the pedestrian action classification and group behavior classification for each pedestrian, respectively.

[0073] In S90, the initial group behavior recognition model is trained by constructing a model loss function based on the outputs of the pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network. The pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network constitute the group behavior recognition model, which is trained using the outputs of each network. The model loss function consists of two parts: a classification loss function corresponding to the pedestrian group classification network and a supervision loss function with a two-branch structure.

[0074] In S100, the group behavior recognition model takes a video clip from a roadside surveillance video as input and outputs the action classification of each pedestrian and the behavior classification of each group in the short video clip. Based on the group behavior recognition of the video clips in the surveillance video, when the recognition results show dangerous behaviors, emergencies, or other specific unsafe factors, the unsafe behaviors are monitored in real time to prevent accidents from occurring.

[0075] The method for group behavior recognition in surveillance video provided by this invention obtains the action and location features of each pedestrian within a continuous time period through a pedestrian action feature network. Further feature extraction is then performed through a temporal action evolution network and a spatial relationship network to obtain the action evolution features of each pedestrian within a continuous time period and the spatial relationship features of all pedestrians in a single video image within a single time period. Behavior represents the actions of a single person or multiple people over a relatively long period, while action represents the actions of a single person within a relatively short period. Furthermore, a dual-branch structure is constructed, first time then space and then space then time, to obtain dual-branch features. This enables spatial feature modeling of multiple pedestrians in each video frame, and also feature modeling of each pedestrian across multiple video frames, allowing for a clearer understanding of the actions of multiple pedestrians. Finally, a pedestrian group classification network classifies the actions of individual pedestrians and groups based on the dual-branch features, resulting in a more comprehensive and accurate understanding of pedestrian behavior in the video.

[0076] Therefore, the group behavior recognition method for surveillance videos provided by this invention enables the recognition of the behavior of each pedestrian within a continuous video sequence, thereby achieving the recognition and understanding of the behavior of multiple people over a period of time. When dangerous behavior or other sudden unsafe behavior is identified, timely warnings can be sent to relevant departments, allowing for the handling of the relevant behavior. This enables intelligent traffic management and public security management, and also avoids dangerous situations occurring late at night or in special road sections with low pedestrian traffic.

[0077] In one embodiment, S20, the pedestrian group training video dataset is input into the pedestrian action feature network to obtain the action features and location features of each pedestrian within a continuous time period, including:

[0078] S210: Input the pedestrian group training video dataset into the convolutional neural network and output the features of each video frame image in a continuous time period;

[0079] S220, based on the features of each video frame image in a continuous time period, pedestrian detection is performed to obtain the action features and position features of each pedestrian in the continuous time period.

[0080] In this embodiment, each video frame image from the video segments in the pedestrian group training video dataset is input into a convolutional neural network, and the output is the image features of each video frame in the video segment. These image features are then input into an object detection network for pedestrian detection, and the output is the action feature representation and the position feature representation of each pedestrian. The object detection network includes, but is not limited to, commonly used pedestrian detection methods such as YOLO, SSD, and CenterNet.

[0081] In one embodiment, S50, spatial relationship features, action features, and location features are fused to obtain a first fused feature, and the first fused feature is input into a temporal action evolution network to obtain a first branch feature, including:

[0082] S510 adds up the spatial relationship features, action features, and position features or splices them together in the channel dimension to obtain the first fused feature.

[0083] In this embodiment, a first fusion feature is obtained by additive processing or feature splicing along the channel dimension. The first fusion feature integrates spatial relationship features, action features, and location features, analyzing the actions of each pedestrian and the behavior of the group from multiple dimensions. This allows for a more accurate understanding of the behavior of each pedestrian in the surveillance video, enabling behavior recognition of each pedestrian within a continuous video sequence, and ultimately achieving the recognition and understanding of the behavior of multiple people over a period of time.

[0084] In one embodiment, S60, the action evolution features, action features, and position features are fused to obtain a second fused feature, and the second fused feature is input into a spatial relationship network to obtain a second branch feature, including:

[0085] S610 adds or splices the action evolution features, action features, and position features in a channel dimension to obtain the second fusion feature.

[0086] In this embodiment, a second fusion feature is obtained by addition or feature splicing along the channel dimension. The second fusion feature integrates action evolution features, action features, and location features, analyzing the actions of each pedestrian and the behavior of the group from multiple dimensions. This allows for a more accurate understanding of the behavior of each pedestrian in the surveillance video, enabling behavior recognition of each pedestrian within a continuous video sequence, and ultimately achieving the recognition and understanding of the behavior of multiple people over a period of time.

[0087] In one embodiment, S90, a model loss function is constructed based on pedestrian action classification, group behavior classification, first branch features, and second branch features; the initial group behavior recognition model is trained to obtain a group behavior recognition model, including:

[0088] S910, construct the cross-entropy classification loss function based on pedestrian action classification and group behavior classification;

[0089] S920, construct a cosine similarity loss function based on the features of the first branch and the features of the second branch;

[0090] S930, the model loss function is formed based on the cross-entropy classification loss function and the cosine similarity loss function.

[0091] In this embodiment, the cross-entropy classification loss function can be used for both the pedestrian action classifier and the group behavior classifier. The cross-entropy classification loss function can be:

[0092]

[0093] Where p represents the category predicted by the pedestrian action classifier or the category predicted by the group behavior classifier, and y=0 and y=1 represent the actual true category as 0 or 1.

[0094] The cosine similarity loss function is a supervised loss function for two branches, which can constrain the outputs of the two branches to be similar. It means that the features output by the space-time branch and the features output by the time-space branch are close to the same, that is, the features of the first branch and the features of the second branch are close to the same.

[0095] The cosine similarity loss function is expressed as:

[0096] L sim =1-cos(a) i ,b i )

[0097] Among them, a i b i These represent the first branch feature and the second branch feature, respectively.

[0098] The model loss function is formed by combining the cross-entropy classification loss function and the cosine similarity loss function. The initial group behavior recognition model is trained to update and optimize the network parameters of the pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network, thereby obtaining a stable group behavior recognition model. The model is then used to perform group behavior recognition on the roadside surveillance video under test, and to obtain the action classification of pedestrians and the behavior classification of the group.

[0099] In one embodiment, the present invention provides a group behavior recognition system 01 for surveillance videos. The group behavior recognition system 01 for surveillance videos includes a pedestrian group data module 10, a pedestrian action feature network module 20, a temporal action evolution network module 30, a spatial relationship network module 40, a first branch module 50, a second branch module 60, a dual-branch feature acquisition module 70, a pedestrian group classification network module 80, a model training module 90, and a behavior acquisition module 100. The pedestrian group data module 10 is used to annotate individual and group actions in roadside surveillance videos to obtain a pedestrian group training video dataset. The pedestrian action feature network module 20 is used to input the pedestrian group training video dataset into the pedestrian action feature network to obtain the action features and location features of each pedestrian within a continuous time period.

[0100] The temporal motion evolution network module 30 is used to input the motion features of each pedestrian within a continuous time period into the temporal motion evolution network to obtain the motion evolution features of each pedestrian within a continuous time period. The spatial relationship network module 40 is used to input the motion features and position features of all pedestrians in a video image within a single time period into the spatial relationship network to obtain the spatial relationship features of all pedestrians in the video image within a single time period. The first branch module 50 is used to fuse the spatial relationship features, motion features, and position features to obtain a first fused feature, and input the first fused feature into the temporal motion evolution network to obtain the first branch feature.

[0101] The second branch module 60 is used to fuse action evolution features, action features, and location features to obtain a second fused feature, which is then input into the spatial relationship network to obtain the second branch feature. The dual-branch feature acquisition module 70 is used to average the first branch feature and the second branch feature to obtain the dual-branch feature. The pedestrian group classification network module 80 is used to input the dual-branch feature into the pedestrian group classification network to obtain pedestrian action classification and group behavior classification.

[0102] The model training module 90 is used to construct a model loss function based on pedestrian action classification, group behavior classification, first branch features, and second branch features, and to train the initial group behavior recognition model to obtain a group behavior recognition model. This model includes a pedestrian action feature network, a temporal action evolution network, a spatial relationship network, and a pedestrian group classification network. The behavior acquisition module 100 is used to perform group behavior recognition on the roadside surveillance video under test based on the group behavior recognition model, obtaining pedestrian action classification and group behavior classification.

[0103] In this embodiment, the description of the pedestrian group data module 10 can be referred to the description of S10 in the above embodiment. The description of the pedestrian action feature network module 20 can be referred to the description of S20 in the above embodiment. The description of the temporal action evolution network module 30 can be referred to the description of S30 in the above embodiment. The description of the spatial relationship network module 40 can be referred to the description of S40 in the above embodiment. The description of the first branch module 50 can be referred to the description of S50 in the above embodiment. The description of the second branch module 60 can be referred to the description of S60 in the above embodiment. The description of the dual-branch feature acquisition module 70 can be referred to the description of S70 in the above embodiment. The description of the pedestrian group classification network module 80 can be referred to the description of S80 in the above embodiment. The description of the model training module 90 can be referred to the description of S90 in the above embodiment. The description of the behavior acquisition module 100 can be referred to the description of S100 in the above embodiment.

[0104] In one embodiment, the pedestrian action feature network module 20 includes an image feature acquisition module and an action position feature acquisition module. The image feature acquisition module is used to input the pedestrian group training video dataset into a convolutional neural network and output the features of each video frame image within a continuous time period. The action position feature acquisition module is used to perform pedestrian detection based on the features of each video frame image within a continuous time period using an object detection network, thereby obtaining the action features and position features of each pedestrian within a continuous time period.

[0105] In this embodiment, the description of the image feature acquisition module can be found in the description of S210 in the above embodiment. The description of the action position feature acquisition module can be found in the description of S220 in the above embodiment.

[0106] In one embodiment, the first branch module includes a first fusion feature acquisition module. The first fusion feature acquisition module is used to sum or concatenate spatial relationship features, action features, and location features in the channel dimension to obtain the first fusion feature.

[0107] In this embodiment, the relevant description of the first fusion feature acquisition module can be referred to the relevant description of S510 in the above embodiment.

[0108] In one embodiment, the second branch module includes a second fusion feature acquisition module. The second fusion feature acquisition module is used to sum or concatenate the action evolution features, action features, and position features along the channel dimension to obtain the second fusion feature.

[0109] In this embodiment, the relevant description of the second fusion feature acquisition module can be referred to the relevant description of S610 in the above embodiment.

[0110] In one embodiment, the model training module includes a first loss function module, a second loss function module, and a model loss function module. The first loss function module constructs a cross-entropy classification loss function based on pedestrian action classification and group behavior classification. The second loss function module constructs a cosine similarity loss function based on first branch features and second branch features. The model loss function module forms the model loss function based on the cross-entropy classification loss function and the cosine similarity loss function.

[0111] In this embodiment, the description of the first loss function module can be found in the description of S910 in the above embodiment. The description of the second loss function module can be found in the description of S920 in the above embodiment. The description of the model loss function module can be found in the description of S930 in the above embodiment.

[0112] In the various embodiments described above, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to a specific order or hierarchy.

[0113] Those skilled in the art will also understand that the various illustrative logical blocks, modules, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, modules, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0114] The various illustrative logic blocks or modules described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0115] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0116] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing group behavior in surveillance video, characterized in that, include: Individual and group behavior annotations were performed on roadside surveillance videos to obtain a training video dataset of pedestrian groups. The pedestrian group training video dataset is input into the pedestrian action feature network to obtain the action features and position features of each pedestrian in a continuous time period. The action features of each pedestrian within the continuous time period are input into the temporal action evolution network to obtain the action evolution features of each pedestrian within the continuous time period; The action features and position features of all pedestrians in a video image within a single time period are input into a spatial relationship network to obtain the spatial relationship features of all pedestrians in the video image within the single time period; The spatial relationship features, the action features, and the position features are fused to obtain a first fused feature, and the first fused feature is input into the temporal action evolution network to obtain a first branch feature; The action evolution feature, the action feature, and the position feature are fused to obtain a second fused feature, and the second fused feature is input into the spatial relationship network to obtain a second branch feature; The first branch feature and the second branch feature are averaged to obtain the bi-branch feature; The dual-branch features are input into the pedestrian group classification network to obtain pedestrian action classification and group behavior classification; A model loss function is constructed based on the pedestrian action classification, the group behavior classification, the first branch feature, and the second branch feature. The initial group behavior recognition model is trained to obtain a group behavior recognition model. The group behavior recognition model includes the pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network. Based on the group behavior recognition model, group behavior recognition is performed on the roadside surveillance video under test to obtain the pedestrian action classification and the group behavior classification.

2. The method for identifying group behavior from surveillance video according to claim 1, characterized in that, The step of inputting the pedestrian group training video dataset into the pedestrian action feature network to obtain the action features and location features of each pedestrian within a continuous time period includes: The pedestrian group training video dataset is input into a convolutional neural network, which outputs the features of each video frame image within the continuous time period. Based on the features of each video frame image within the continuous time period, pedestrian detection is performed using the target detection network to obtain the action features and position features of each pedestrian within the continuous time period.

3. The method for identifying group behavior from surveillance video according to claim 1, characterized in that, The process of fusing the spatial relationship features, the action features, and the position features to obtain a first fused feature, and inputting the first fused feature into the temporal action evolution network to obtain a first branch feature, includes: The first fused feature is obtained by summing the spatial relationship features, the action features, and the position features, or by splicing them together along the channel dimension.

4. The method for identifying group behavior from surveillance video according to claim 1, characterized in that, The process of fusing the action evolution features, the action features, and the position features to obtain a second fused feature, and then inputting the second fused feature into the spatial relationship network to obtain a second branch feature, includes: The second fused feature is obtained by summing the action evolution feature, the action feature, and the position feature or by splicing them together in the channel dimension.

5. The method for identifying group behavior from surveillance video according to claim 1, characterized in that, The step of constructing a model loss function based on the pedestrian action classification, the group behavior classification, the first branch features, and the second branch features, and training the initial group behavior recognition model to obtain the group behavior recognition model includes: Based on the pedestrian action classification and the group behavior classification, a cross-entropy classification loss function is constructed; Based on the features of the first branch and the features of the second branch, a cosine similarity loss function is constructed; The model loss function is formed based on the cross-entropy classification loss function and the cosine similarity loss function.

6. A group behavior recognition system for surveillance video, characterized in that, include: The pedestrian group data module is used to annotate individual and group behaviors in roadside surveillance videos to obtain a pedestrian group training video dataset. The pedestrian action feature network module is used to input the pedestrian group training video dataset into the pedestrian action feature network to obtain the action features and position features of each pedestrian in a continuous time period. The temporal action evolution network module is used to input the action features of each pedestrian within the continuous time period into the temporal action evolution network to obtain the action evolution features of each pedestrian within the continuous time period. The spatial relationship network module is used to input the action features and position features of all pedestrians in the video image within a single time period into the spatial relationship network to obtain the spatial relationship features of all pedestrians in the video image within the single time period; The first branch module is used to fuse the spatial relationship features, the action features, and the position features to obtain a first fused feature, and input the first fused feature into the time-action evolution network to obtain a first branch feature; The second branch module is used to fuse the action evolution features, the action features, and the position features to obtain a second fused feature, and input the second fused feature into the spatial relationship network to obtain a second branch feature; The dual-branch feature acquisition module is used to average the first branch feature and the second branch feature to obtain the dual-branch feature. The pedestrian group classification network module is used to input the dual-branch features into the pedestrian group classification network to obtain pedestrian action classification and group behavior classification; The model training module is used to construct a model loss function based on the pedestrian action classification, the group behavior classification, the first branch feature, and the second branch feature, and to train the initial group behavior recognition model to obtain a group behavior recognition model. The group behavior recognition model includes the pedestrian action feature network, the temporal action evolution network, the spatial relationship network, and the pedestrian group classification network. The behavior acquisition module is used to perform group behavior recognition on the roadside surveillance video under test according to the group behavior recognition model, and obtain the action classification of pedestrians and the behavior classification of the group.

7. The group behavior recognition system for surveillance video according to claim 6, characterized in that, The pedestrian action feature network module includes: The image feature acquisition module is used to input the pedestrian group training video dataset into the convolutional neural network and output the features of each video frame image in the continuous time period; The action location feature acquisition module is used to perform pedestrian detection based on the features of each video frame image within the continuous time period using the target detection network, and to obtain the action features and location features of each pedestrian within the continuous time period.

8. The group behavior recognition system for surveillance video according to claim 6, characterized in that, The first branch module includes: The first fusion feature acquisition module is used to sum or concatenate the spatial relationship features, the action features, and the position features in the channel dimension to obtain the first fusion feature.

9. The group behavior recognition system for surveillance video according to claim 6, characterized in that, The second branch module includes: The second fusion feature acquisition module is used to sum or concatenate the action evolution feature, the action feature, and the position feature in the channel dimension to obtain the second fusion feature.

10. The group behavior recognition system for surveillance video according to claim 6, characterized in that, The model training module includes: The first loss function module is used to construct a cross-entropy classification loss function based on the pedestrian action classification and the group behavior classification. The second loss function module is used to construct a cosine similarity loss function based on the first branch features and the second branch features; The model loss function module is used to form the model loss function based on the cross-entropy classification loss function and the cosine similarity loss function.

Images