A self-supervised group behavior recognition method based on spatiotemporal serial-parallel relation coding

Abstract

This application discloses a self-supervised group behavior recognition method based on spatiotemporal serial-parallel relationship coding, belonging to the field of video analysis technology. The method includes acquiring group behavior video data; constructing a network structure for a group feature self-learning model based on spatiotemporal serial-parallel attention mechanism relationship prediction coding, the network structure including a group label generator, a serial-parallel Transformer encoder, and an attention mechanism decoder; performing self-supervised training on the network structure to obtain a feature self-learning target network; and fine-tuning the feature self-learning target network based on a small number of samples to obtain a target network for realizing group behavior recognition. This application fully explores the spatial coordination relationships and temporal dynamic changes in the group through spatiotemporal serial-parallel attention mechanism relationship prediction coding, thereby improving the ability to construct state transitions and learn the expression of complex group features. It is suitable for efficient and accurate group behavior recognition in cases lacking annotations.

Description

TECHNICAL FIELD

[0001] The application relates to a self-supervised group behavior recognition method based on spatiotemporal serial-parallel relationship prediction coding, and belongs to the technical field of video analysis. BACKGROUND

[0002] The main purpose of group behavior recognition is to identify the activities performed by a group of people in a scene, and its application fields include monitoring analysis, sports video understanding and military strategy analysis. In recent years, group behavior recognition has attracted great attention from the academic and industrial circles due to its great application potential and scientific challenges. So far, the method based on a deep learning model has replaced the traditional feature extraction method. However, although the existing group behavior recognition technology based on the deep learning algorithm has made significant progress, it is mostly based on supervised learning or weakly supervised learning, and needs to use a large number of labeled data sets to train the model, that is, relies on manually labeled group behavior labels, which is costly and needs to consume a large amount of manpower and resources.

[0003] In view of the above problems, some current researches focus on self-supervised learning (SSL). Generally, the existing SSL method is usually a time-based method, which is mainly used for sequence data representation learning, such as video, which has guiding significance for group behavior analysis. Contrastive learning is a self-supervised representation learning method. In recent years, with the great success of contrastive learning, many methods have been used to extract video feature representation, in addition, the prediction coding scheme using contrastive learning for future prediction has attracted more and more attention in video understanding, such as dense prediction coding (DPC) and memory-enhanced dense prediction coding (MemDPC).

[0004] However, the above SSL method still has some difficulties in group behavior representation learning. The current video SSL method based on contrastive learning usually adopts overall representation, for example, the video features extracted by a 3D CNN are used to construct a representation pair, which will limit the performance due to the lack of model interaction. In addition, the individual interaction in group activities changes greatly over time and lasts for a longer time. The existing video SSL method realizes the spatiotemporal consistency of the representation pair from a large time interval, which can only capture little shared information, while the prediction coding scheme considers the time evolution by predicting the future change to a consistent space to maximize the consistency between the contrastive representation pairs, which shows the ability to learn group behavior representation. However, the above methods are limited by the insufficient ability to describe complex relationship conversion, which will greatly affect the understanding of group behavior. Therefore, it is of great significance to improve the performance of group behavior recognition by predicting the spatiotemporal relationship by comprehensively considering the spatial individual interaction and the temporal change. SUMMARY

[0005] The application aims to provide a self-supervised group behavior recognition method based on space-time series-parallel relationship coding, a space-time series-parallel attention mechanism relationship prediction coding, a combination of spatial interaction and time variation to construct a series-parallel Transformer encoder, and then a new end-to-end group feature self-learning model to realize efficient and accurate group behavior recognition under the condition of lacking annotation.

[0006] To achieve the above-mentioned purpose, the first aspect of the application provides a self-supervised group behavior recognition method based on space-time series-parallel relationship coding, comprising:

[0007] Obtaining group behavior video data;

[0008] Based on the space-time series-parallel attention mechanism relationship prediction coding, the network structure of the group feature self-learning model is constructed, and the network structure comprises a group label generator, a series-parallel Transformer encoder and an attention mechanism decoder, wherein the group label generator generates a group label for initializing individual representation according to the group behavior video data; the series-parallel Transformer encoder fuses spatial information and time information according to the group label and generates high-level semantics for constructing group relationship; and the attention mechanism decoder predicts the group behavior space-time characteristics of future frames according to the high-level semantics.

[0009] The network structure is self-supervised trained to obtain a feature self-learning target network;

[0010] The feature self-learning target network is fine-tuned based on a small amount of samples to obtain a target network, and the self-supervised group behavior recognition is performed through the target network.

[0011] In an implementation mode, the group label generator is specifically used for:

[0012] The pre-trained I3D network is used as a backbone to extract feature maps for feature extraction of the input video segment;

[0013] According to the results of feature extraction, an appearance feature branch and a posture information branch are respectively established;

[0014] The appearance feature branch and the posture information branch are connected through a full connection layer to obtain a D-dimensional feature vector of each individual in the video segment;

[0015] The D-dimensional feature vectors are combined to obtain the group label.

[0016] In an embodiment, the serial-parallel Transformer encoder comprises a plurality of spatial attention blocks and a plurality of temporal attention blocks, and each of the spatial attention blocks and each of the temporal attention blocks is designed in a twisted fusion manner, wherein the spatial attention blocks are used to obtain inter-individual dependency in the t-th frame of the video clip, and the temporal attention blocks are used to obtain dynamic information of the same individual in the frame sequence of the video clip.

[0017] The serial-parallel Transformer encoder is specifically used for:

[0018] The spatial attention blocks and the temporal attention blocks are operated in a parallel manner to project spatial group behavior features according to the inter-individual dependency in the t-th frame of the video clip and to project temporal group behavior features according to the dynamic information of the same individual in the frame sequence of the video clip.

[0019] The embedded spatial group behavior features and the temporal group behavior features are added after being converted to the same size and are transmitted to other spatial attention blocks and temporal attention blocks in a serial manner for integration to obtain spatial high-level semantic encoding information and temporal high-level semantic encoding information as the high-level semantics.

[0020] In an embodiment, the attention mechanism decoder is specifically used for:

[0021] The spatial high-level semantic encoding information and the temporal high-level semantic encoding information are combined through a self-attention mechanism, and then the group behavior spatiotemporal features of each future frame are gradually predicted by a bidirectional prediction scheme through convolution, normalization and pooling operations.

[0022] In an embodiment, the self-supervised training of the network structure comprises:

[0023] In the process of prediction by the attention mechanism decoder through the bidirectional prediction scheme, a joint loss is calculated based on the predicted group behavior spatiotemporal features.

[0024] Based on the joint loss, each parameter of the network structure is iteratively adjusted by a loss gradient backpropagation algorithm until the training converges, and the feature self-learning target network is obtained.

[0025] In an embodiment, the joint loss comprises:

[0026] A pixel-level reconstruction loss, an inter-group contrast loss and an adversarial loss, wherein the pixel-level reconstruction loss is used to improve the prediction accuracy at a pixel-level fine granularity, the inter-group contrast loss is used to improve the authenticity and accuracy of group-level prediction at a coarse granularity, and the adversarial loss is used to improve the continuity of the predicted sequence.

[0027] In an implementation, the fine-tuning of the feature self-learning target network based on the small amount of samples includes:

[0028] According to a preset group behavior recognition task, the feature self-learning target network is fine-tuned based on a small amount of supervised learning samples, so as to evaluate the representation self-learning ability of the feature self-learning target network.

[0029] In the group behavior recognition task of fine-tuning the entire network structure, the fine-tuning of the feature self-learning target network based on the small amount of samples includes: adding a linear classification layer after the feature self-learning target network, and fine-tuning all parameters of the feature self-learning target network and the linear classification layer under the constraint of the labeled training samples.

[0030] The second aspect of the application provides a self-supervised group behavior recognition device based on spatiotemporal series-parallel relationship coding, comprising:

[0031] An acquisition module is configured to acquire group behavior video data.

[0032] A construction module is configured to construct a network structure of a group feature self-learning model based on spatiotemporal series-parallel attention mechanism relationship prediction coding, wherein the network structure includes a group label generator, a series-parallel Transformer encoder, and an attention mechanism decoder.

[0033] A training module is configured to perform self-supervised training on the network structure to obtain a feature self-learning target network.

[0034] A fine-tuning module is configured to fine-tune the feature self-learning target network based on a small amount of samples to obtain a target network, and perform self-supervised group behavior recognition through the target network.

[0035] The third aspect of the application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the first aspect or any of the implementation manners of the first aspect.

[0036] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the first aspect or any one of the implementation manners of the first aspect.

[0037] As can be seen from the above, the present application provides a self-supervised group behavior recognition method based on spatiotemporal series-parallel relationship prediction coding, which comprises: acquiring group behavior video data; constructing a network structure of a group feature self-learning model based on a spatiotemporal series-parallel attention mechanism relationship prediction coding, the network structure comprising a group label generator, a series-parallel Transformer encoder and an attention mechanism decoder; performing self-supervised training on the network structure to obtain a feature self-learning target network; and fine-tuning the feature self-learning target network based on a small amount of samples to obtain a target network for realizing group behavior recognition. The present application provides a new end-to-end framework, called a group behavior feature self-learning model based on a spatiotemporal series-parallel attention mechanism relationship prediction coding, for group behavior representation self-learning. The spatiotemporal series-parallel attention mechanism relationship prediction coding fully excavates the spatial coordination relationship and the temporal dynamic change in the group, thereby improving the construction ability of state transformation and learning the expression of complex group features. The present application is suitable for efficient and accurate group behavior recognition under the condition of lacking labels. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0039] Figure 1 A flowchart of a network structure of a group feature self-learning model provided by the embodiments of the present application is shown in the figure.

[0040] Figure 2 A flowchart of fine-tuning the feature self-learning target network provided by the embodiments of the present application is shown in the figure.

[0041] Figure 3 An effect display figure of self-supervised group behavior recognition by the target network provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0042] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0043] It is to be understood that the terminology "including", "comprising", "consisting" and "consisting essentially of" used in the specification and the appended claims, indicates the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0044] It is also to be understood that the terminology used in the present specification and the appended claims is for the purpose of describing the particular embodiments only and is not intended to be limiting.

[0045] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0046] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0047] Embodiment one

[0048] The embodiments of the present application provide a self-supervised group behavior recognition method, as shown in Figure 1 and Figure 2 The method comprises:

[0049] obtaining group behavior video data;

[0050] The network structure of the group feature self-learning model is constructed based on a spatiotemporal series-parallel attention mechanism relationship prediction coding, and the network structure comprises a group label generator, a series-parallel Transformer encoder and an attention mechanism decoder, wherein the group label generator generates a group label for initializing individual representation according to the group behavior video data; the series-parallel Transformer encoder extracts spatial information and temporal information according to the group label and generates high-level semantics for constructing group relationship; and the attention mechanism decoder predicts the group behavior spatiotemporal features of future frames according to the high-level semantics.

[0051] The network structure is subjected to self-supervised training to obtain a feature self-learning target network.

[0052] The feature self-learning target network is fine-tuned based on a small amount of samples to obtain a target network, and the target network is used for self-supervised group behavior recognition.

[0053] In an embodiment, the group behavior video data comprises data in two data sets of Volleyball Dataset and Collective Activity Dataset; the group behavior video data is preprocessed respectively, and all video frames are scaled to a resolution of 720x1280; T=10 frames are selected as inputs of the group feature self-learning model for training in subsequent steps, so as to realize analysis of group behavior on multiple video data sets.

[0054] Optionally, the group label generator is specifically configured to:

[0055] A pre-trained I3D network is used as a backbone to extract feature maps for feature extraction of the input video segment;

[0056] An appearance feature branch and a posture information branch are respectively established according to the feature extraction result;

[0057] A D-dimensional feature vector of each individual in the video segment is obtained by connecting the appearance feature branch and the posture information branch through a fully connected layer;

[0058] The D-dimensional feature vectors are combined to obtain the group label.

[0059] In an embodiment, the group label generator is a pre-processing module containing rich individual information from a video to realize initialization of individual representation. Specifically, given a T-frame input video segment X video Since the position of the body joint and its movement determine the action of the individual, the pre-trained I3D network is first used as a backbone to extract feature maps for X videofeature extraction; the group label generator establishes two branches using appearance features and pose information, and further connects them into a D-dimensional feature vector for each person through a fully connected layer. Finally, these individual D-dimensional feature vectors are combined into the final group label X G .

[0060] Optionally, the serial-parallel Transformer encoder comprises a plurality of spatial attention blocks and temporal attention blocks, and each of the spatial attention blocks and each of the temporal attention blocks is designed in a twisted fusion manner, wherein the spatial attention block is used to obtain inter-individual dependency in the t-th frame of the video clip, and the temporal attention block is used to obtain dynamic information of the same individual in the frame sequence of the video clip:

[0061] The serial-parallel Transformer encoder is specifically used for:

[0062] The spatial attention block and the temporal attention block are run in a parallel manner to project a spatial group behavior feature according to the inter-individual dependency in the t-th frame of the video clip and project a temporal group behavior feature according to the dynamic information of the same individual in the frame sequence of the video clip;

[0063] The embedded spatial group behavior feature and the temporal group behavior feature are converted to the same size and then added, and are transmitted to other spatial attention blocks and temporal attention blocks in a serial manner for integration to obtain spatial high-level semantic encoding information and temporal high-level semantic encoding information.

[0064] In an embodiment, the purpose of the serial-parallel Transformer encoder is to construct the group relationship in the observation X video In order to emphasize the extraction and fusion of multi-view semantics, the serial-parallel Transformer encoder comprises a plurality of serial-parallel spatial attention blocks and temporal attention blocks and is designed in a twisted fusion manner. The spatial attention block is performed on the input group label X G generated by the group label generator to capture the inter-individual dependency of N participants in the t = {1, 2, ···, T} frame. The serial-parallel Transformer encoder is based on the Transformer structure, and the index value of the key K corresponds to other participants in each frame. The dot product similarity of the query Q and the corresponding key K is calculated as a weight representing the degree of attention to other individuals. Then, the attention operation generates a weighted sum of the value V, and then uses a fully connected layer to generate the input X G,t to Q, K, V matrix using multi-head attention. The output of the multi-head attention is followed by a normalization layer (Norm) and a feedforward network (FFN) to generate a feature (i.e., a spatial group behavior feature) representation in the t-th frame, represented as The feature representation of all T frames can be represented as This is the final output of the spatial attention block. The spatial attention block focuses on the individual correlation in each frame, while the temporal attention block extracts the dynamic information of the same individual in the frame sequence. For each individual n∈{1,2…,N}, given the embedding of the action in the history record, the sequence correlation is generated by multi-head attention. The spatial dimension N in X G is regarded as the batch size, where each vector can be regarded as the historical dynamics of an individual. In the temporal attention block, the output of the multi-head attention is processed using addition, normalization and fully connected feedforward network to obtain the final temporal variation (i.e. temporal group behavior feature) representation, denoted as The spatial attention block and the temporal attention block are fused in a twisted way, i.e. the spatial attention and the temporal attention are interleaved in a single framework, so as to fully integrate the spatial information and the temporal information. Specifically, the spatial and temporal attention blocks are first run in parallel to project the spatial group behavior features and the temporal group behavior features, respectively. Then, the embeddings are converted to the same size and added, and are passed to other spatial attention blocks and temporal attention blocks in a serial manner to further integrate the temporal advanced semantic encoding information and the spatial advanced semantic encoding information.

[0065] Optionally, the attention mechanism decoder is specifically configured to:

[0066] The spatial advanced semantic encoding information and the temporal advanced semantic encoding information are combined through a self-attention mechanism, and then the group behavior spatio-temporal features of each future frame are gradually predicted by a bidirectional prediction scheme through convolution, normalization and pooling operations.

[0067] In an embodiment, the attention mechanism decoder combines the spatial and temporal advanced semantics through a self-attention mechanism to gradually predict the group behavior spatio-temporal features of each future frame by a bidirectional prediction scheme. Specifically, the advanced spatial semantic features generated from the serial spatial encoder in the serial-parallel Transformer encoder are regarded as the query Q, and the advanced temporal semantic features generated from the serial temporal decoder are regarded as the key value K, V. In each prediction step, the relationship semantic information captures the group relationship features from K by calculating the similarity of Q and K. The attention operation produces the output of the weighted sum of V, where Q is further added to update the self-attention. The prediction result of the t+1 frame can be represented as as an output of the attention mechanism decoder. In group behavior, since time is continuous, the forward relationship change should be consistent with the backward prediction, so a bidirectional prediction scheme is applied in the prediction step, and the observed video frame and the future video frame are respectively taken as input to perform bidirectional spatio-temporal feature prediction, thereby improving the consistency of feature self-learning. Alternatively, in other embodiments, the attention mechanism decoder can also predict the group behavior features in other ways, which are not limited here.

[0068] Optionally, the self-supervised training of the network structure comprises:

[0069] In the process of predicting by the attention mechanism decoder through the bidirectional prediction scheme, a joint loss is calculated based on the predicted group behavior spatio-temporal features;

[0070] Based on the joint loss, each parameter of the network structure is iteratively adjusted by a loss gradient backpropagation algorithm until the training converges, and the feature self-learning target network is obtained.

[0071] Optionally, the joint loss comprises:

[0072] Pixel-level reconstruction loss, inter-group contrast loss, and adversarial loss, wherein the pixel-level reconstruction loss is used to improve the prediction accuracy at the pixel level; the inter-group contrast loss is used to improve the authenticity and accuracy of the group-level prediction at the coarse granularity; and the adversarial loss is used to improve the continuity of the predicted sequence.

[0073] In an embodiment, the pixel-level reconstruction loss promotes the accuracy of details of the reconstructed group behavior video, i.e., ensures the consistency of the pixel values of each predicted frame with the pixel values of the original video frame, and ensures the accuracy of the color, lighting, and overall contrast of each video frame, thereby improving the prediction accuracy at the pixel level. It can be described as follows:

[0074]

[0075] wherein, is a real group activity video frame, is a corresponding predicted reconstructed video frame. The pixel-level reconstruction loss is the sum of the L1 losses of the pixel values of each frame of the real video and the predicted reconstructed video.

[0076] Further, contrast learning is to construct positive and negative sample pairs by calculating similarity, so as to force the score of the positive sample pair to be higher than that of the negative sample pair. Therefore, in the inter-group contrast loss, the positive sample pair refers to the same position of the predicted and extracted group features (i.e., group behavior spatio-temporal features), and the negative sample pair comes from different time positions. The objective function to be minimized is:

[0077]

[0078] where i, j denote time indices, and φ(·) is the dot product that computes the similarity between two vectors. p is X p pooling the obtained group features, is the corresponding extracted features. This group optimization can improve the authenticity and accuracy of the coarse-grained group-level prediction.

[0079] Further, the adversarial loss is used to improve the coherence of the predicted sequence. The predicted group features of the future T' frames in the attention mechanism decoder are regarded as fake samples, and the feature descriptors extracted from the T-frame video are regarded as real samples. Based on these real / fake sample pairs, the generator G and the discriminator D are in confrontation, so that the framework generates sequences that are difficult to distinguish from real samples, to improve the coherence of the predicted sequence. This can be described as follows:

[0080]

[0081] where the distribution E(·) is on the training sequence. The discriminator D is a classic Transformer structure followed by a pooling and linear classification layer, which outputs the probability of being real.

[0082] Further, the sum of the above three parts of the loss is the joint loss, denoted as:

[0083]

[0084] Optionally, since it is not possible to directly evaluate the representation learning ability of the group feature self-learning model, the embodiments of the present application perform fine-tuning of the downstream task based on supervised learning on the feature self-learning target network obtained based on the above self-supervised training, and perform evaluation, as shown in Figure 2 The fine-tuning of the feature self-learning target network based on a small number of samples includes:

[0085] According to the preset group behavior recognition task, the feature self-learning target network is fine-tuned based on a small number of supervised learning samples to realize evaluation of the representation self-learning ability of the feature self-learning target network.

[0086] Wherein, the group behavior recognition task is to fine-tune the entire network structure for group behavior recognition, and the fine-tuning of the feature self-learning target network based on a small number of samples includes: adding a linear classification layer after the feature self-learning target network, and all parameters of the feature self-learning target network and the linear classification layer are fine-tuned under the constraint of labeled training samples.

[0087] In an embodiment, in the supervised learning of group behavior recognition, the input group behavior video sample follows the same sampling procedure as the above-mentioned group feature self-learning model. At this time, in the training stage, all frames in the video sample can be used for training without considering prediction. And when the group behavior recognition is evaluated with a preset group behavior recognition task, the cross-entropy loss can be used to train the classifier. Specifically, in the training of the linear classifier for group behavior recognition, all samples in the training set are input into the network to train the classifier; in the fine-tuning of the entire network structure for group behavior recognition, 10% of the labeled samples are randomly selected to fine-tune all parameters in the network structure. In the test stage, the standard process is followed to obtain the same sequence length as in the training stage for testing, and then the group behavior is analyzed on the video dataset.

[0088] As can be seen from the above, the embodiment of the application provides a self-supervised group behavior recognition method based on spatiotemporal series-parallel relationship prediction coding, which comprises: acquiring group behavior video data; constructing a network structure of a group feature self-learning model based on spatiotemporal series-parallel attention mechanism relationship prediction coding, the network structure comprising a group label generator, a series-parallel Transformer encoder and an attention mechanism decoder; performing self-supervised training on the network structure to obtain a feature self-learning target network; and fine-tuning the feature self-learning target network based on a small amount of samples to evaluate the representation self-learning ability of the feature self-learning target network. The embodiment of the application provides a new end-to-end framework, called a group behavior feature self-learning model based on spatiotemporal series-parallel attention mechanism relationship prediction coding, for group behavior representation learning. The encoder is based on the Transformer structure and fully excavates the spatial coordination relationship and temporal dynamic change in the group through spatiotemporal series-parallel attention mechanism relationship prediction coding, and fuses spatial information and temporal information, thereby improving the construction ability of state transformation, generating high-level semantics for constructing complex group relationships, and obtaining more comprehensive self-supervised group behavior representation.

[0089] Embodiment Two

[0090] The effect of the self-supervised group behavior recognition method provided in Embodiment One of the application is demonstrated below through an experiment.

[0091] The self-supervised collective behavior recognition method based on the embodiments of the present application is tested on two widely used collective behavior datasets (a Volleyball Dataset and a Collective Activity Dataset), and compared with the latest method of all datasets. The experimental results show that the self-supervised collective behavior recognition method proposed in the embodiments of the present application has higher recognition accuracy than the existing method, and confirms the effectiveness and generality of the method. The effect display diagram of the self-supervised collective behavior recognition method based on the spatiotemporal series-parallel attention mechanism relationship prediction encoding is as shown in Figure 3 The real value is the collective activity label of the video, and the recognition result is the collective activity description recognized based on the collective behavior recognition method. As can be seen from the figure, the collective behavior recognition method proposed in the embodiments of the present application has superiority in self-supervised collective behavior representation learning.

[0092] Embodiment three

[0093] The embodiments of the present application provide a self-supervised collective behavior recognition device based on spatiotemporal series-parallel relationship encoding, which comprises an acquisition module, a construction module, a training module and a fine-tuning module.

[0094] The acquisition module is configured to acquire collective behavior video data.

[0095] The construction module is configured to construct the network structure of the collective feature self-learning model based on the spatiotemporal series-parallel attention mechanism relationship prediction encoding. The network structure comprises a collective label generator, a series-parallel Transformer encoder and an attention mechanism decoder. The collective label generator generates a collective label for initializing individual representation according to the collective behavior video data. The series-parallel Transformer encoder fuses spatial information and temporal information and generates high-level semantics for constructing collective relationship according to the collective label. The attention mechanism decoder predicts the collective behavior spatiotemporal features of future frames according to the high-level semantics.

[0096] The training module is configured to perform self-supervised training on the network structure to obtain a feature self-learning target network.

[0097] The fine-tuning module is configured to fine-tune the feature self-learning target network based on a small amount of samples to obtain a target network, and perform self-supervised collective behavior recognition through the target network.

[0098] As can be seen, the embodiment of the present application provides a self-supervised group behavior recognition device based on spatiotemporal series-parallel relationship prediction coding, which is based on a Transformer structure, fully mines the spatial coordination relationship and temporal dynamic change in the group through spatiotemporal series-parallel attention mechanism relationship prediction coding, fuses spatial information and temporal information, thereby improving the construction ability of state transformation, generating high-level semantics for constructing complex group relationship, and obtaining more comprehensive self-supervised group behavior representation.

[0099] Embodiment four

[0100] The embodiment of the present application provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the memory is used for storing software programs and modules, and the processor executes various function applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected through a bus. Specifically, the processor realizes any step in the above embodiment one when running the above computer program stored in the memory.

[0101] It should be understood that, in the embodiment of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0102] The memory can include read-only memory, flash memory and random access memory, and provide instructions and data for the processor. Part or all of the memory can also include non-volatile random access memory.

[0103] As can be seen, the electronic device provided by the embodiment of the present application can realize the self-supervised group behavior recognition method as described in the embodiment one by running the computer program, obtain a new end-to-end framework, called group feature self-learning model, for group behavior representation learning. The spatiotemporal series-parallel attention mechanism relationship prediction coding fully mines the spatial coordination relationship and temporal dynamic change in the group, thereby improving the construction ability of state transformation, learning the expression of complex group features, and thereby realizing high-precision group behavior recognition under lack of annotation.

[0104] It should be understood that the above-mentioned integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the above-mentioned computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The above-mentioned computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The above-mentioned computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, ROM), electric carrier signal, telecommunication signal and software distribution medium, etc. capable of carrying the above-mentioned computer program code. It should be noted that the content contained in the above-mentioned computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0105] The above description of disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software function unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above-mentioned system can refer to the corresponding process in the above-mentioned method embodiments, which will not be repeated here.

[0107] It should be noted that the method and details thereof provided by the above embodiments can be combined with the apparatus and device provided by the embodiments, and mutual reference is not repeated.

[0108] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0109] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the above-described apparatus / device embodiments are merely illustrative, for example, the division of the above-described modules or units is merely a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0110] The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A self-supervised group behavior recognition method based on spatiotemporal serial-parallel relationship coding, characterized in that, The method comprises the following steps: acquiring group behavior video data; constructing a network structure of a group feature self-learning model based on a spatiotemporal serial-parallel attention mechanism relationship prediction coding, the network structure comprising a group label generator, a serial-parallel Transformer encoder and an attention mechanism decoder, wherein the group label generator generates a group label for initializing individual representation according to the group behavior video data; the serial-parallel Transformer encoder fuses spatial information and temporal information according to the group label and generates high-level semantics for constructing group relationship; and the attention mechanism decoder predicts group behavior spatiotemporal features of future frames according to the high-level semantics; performing self-supervised training on the network structure to obtain a feature self-learning target network; fine-tuning the feature self-learning target network based on a small amount of samples to obtain a target network, and performing self-supervised group behavior recognition through the target network; the serial-parallel Transformer encoder comprises a plurality of spatial attention blocks and a plurality of temporal attention blocks, and each spatial attention block and each temporal attention block are designed in a twisted fusion manner, wherein the spatial attention block is used to obtain the inter-individual dependency relationship in the t-th frame of the video segment, and the temporal attention block is used to obtain the dynamic information of the same individual in the frame sequence of the video segment; the serial-parallel Transformer encoder is specifically used for: running the spatial attention block and the temporal attention block in a parallel manner to project spatial group behavior features according to the inter-individual dependency relationship in the t-th frame of the video segment and to project temporal group behavior features according to the dynamic information of the same individual in the frame sequence of the video segment; adding the embedded spatial group behavior features and the temporal group behavior features after converting them to the same size, and transmitting them to other spatial attention blocks and temporal attention blocks in a serial manner for integration to obtain spatial high-level semantic coding information and temporal high-level semantic coding information as the high-level semantics.

2. The self-supervised group behavior recognition method of claim 1, wherein, The group label generator is specifically used for: extracting features of the input video segment by taking a pre-trained I3D network as a backbone to extract feature maps; establishing an appearance feature branch and a posture information branch respectively according to the feature extraction result; connecting the appearance feature branch and the posture information branch through a fully connected layer to obtain a D-dimensional feature vector of each individual in the video segment; combining the D-dimensional feature vectors to obtain the group label.

3. The self-supervised group behavior recognition method of claim 1, wherein, The attention mechanism decoder is specifically used for: combining the spatial high-level semantic coding information and the temporal high-level semantic coding information through a self-attention mechanism, and then gradually predicting the group behavior spatiotemporal features of each future frame through convolution, normalization and pooling operations by a bidirectional prediction scheme.

4. The self-supervised group behavior recognition method of claim 3, wherein, The self-supervised training of the network structure comprises: calculating a joint loss based on the predicted group behavior spatiotemporal features during the prediction of the attention mechanism decoder by the bidirectional prediction scheme. Based on the joint loss, each parameter of the network structure is iteratively adjusted by a loss gradient back propagation algorithm until training converges, and the feature self-learning target network is obtained.

5. The self-supervised group behavior recognition method of claim 4, wherein, The joint loss includes: The pixel-level reconstruction loss, the inter-group contrast loss, and the adversarial loss, wherein the pixel-level reconstruction loss is used to improve the prediction accuracy at the pixel level; the inter-group contrast loss is used to improve the authenticity and accuracy of the group-level prediction at the coarse granularity; and the adversarial loss is used to improve the continuity of the predicted sequence.

6. The self-supervised group behavior recognition method of claim 1, wherein, The fine-tuning of the feature self-learning target network based on a small amount of samples includes: According to a preset group behavior recognition task, the feature self-learning target network is fine-tuned based on a small amount of supervised learning samples, so as to evaluate the representation self-learning ability of the feature self-learning target network. Wherein, when fine-tuning the entire network structure for group behavior recognition, the fine-tuning of the feature self-learning target network based on a small amount of samples includes: adding a linear classification layer after the feature self-learning target network, and fine-tuning all parameters of the feature self-learning target network and the linear classification layer under the constraint of labeled training samples.

7. A self-supervised group behavior recognition device based on spatiotemporal serial-parallel relationship coding, characterized in that, The identification device is used to implement the self-supervised group behavior recognition method based on the space-time series-parallel relationship coding according to any one of claims 1-6, and the identification device includes: An acquisition module is configured to acquire group behavior video data. A construction module is configured to construct a network structure of a group feature self-learning model based on space-time series-parallel attention mechanism relationship prediction coding, wherein the network structure includes a group label generator, a series-parallel Transformer encoder, and an attention mechanism decoder, wherein the group label generator generates a group label for initializing individual representation according to the group behavior video data; the series-parallel Transformer encoder fuses spatial information and temporal information and generates high-level semantics for constructing group relationship according to the group label; and the attention mechanism decoder predicts future frame group behavior space-time features according to the high-level semantics. A training module is configured to perform self-supervised training on the network structure to obtain a feature self-learning target network. A fine-tuning module is configured to fine-tune the feature self-learning target network based on a small amount of samples to obtain a target network, and perform self-supervised group behavior recognition through the target network.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-6.

Images