A self-supervised group behavior recognition method based on context-aware relation prediction coding

Abstract

This application discloses a self-supervised group behavior recognition method based on context-aware relation prediction coding, belonging to the field of video analysis technology. The method includes: acquiring group behavior video data; constructing a network structure for a self-learning model of group behavior features based on context-aware relation prediction coding, the network structure including a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder; performing self-supervised training on the network structure to obtain a target network for self-learning group behavior features; and fine-tuning the target network based on a small number of samples to obtain a target network for achieving group behavior recognition. This application captures contextual relationships through context-aware relation prediction coding, thereby obtaining a more comprehensive and accurate self-learning representation of group behavior. It is suitable for efficient and accurate group behavior recognition in situations lacking annotations.

Description

Technical Field

[0001] This application relates to a self-supervised group behavior recognition method based on context-aware relation prediction coding, belonging to the field of video analytics technology. Background Technology

[0002] Group behavior recognition refers to the process of identifying group activities performed by multiple individuals in a video containing a multi-person scene. It is a crucial problem in video understanding, applicable to real-world scenarios such as sports video analysis, surveillance video recognition, and social behavior understanding. Due to its wide range of applications, group behavior recognition has attracted significant attention from researchers. Currently, with the continuous development of deep learning, feature extraction methods based on deep neural networks have replaced traditional algorithms. However, while existing deep learning-based group behavior recognition technologies have made significant progress, most rely on supervised or weakly supervised learning. This requires training models using large amounts of labeled datasets, and manually labeling group behaviors consumes substantial manpower and resources, resulting in high costs.

[0003] To address the aforementioned issues, current research focuses on self-supervised learning (SSL). Generally, existing SSL methods are context-based and are also known as self-supervised representation learning for images. Contrastive learning is a type of self-supervised representation learning method. In recent years, with its significant success, many methods have applied it to extract video feature representations. Furthermore, encoding / decoding schemes utilizing contrastive learning for video prediction have received increasing attention in video understanding, such as Contrastive Action Representation Learning (CARL) and Co-trained Self-Supervised Contrastive Learning (CoCLR).

[0004] However, due to the complex contextual dynamics in group behavior, the aforementioned SSL methods still face some challenges in learning representations of group behavior. Current video SSL methods based on contrastive learning typically employ holistic representations, such as constructing representation pairs using video features extracted by 3D CNNs, which limits performance due to insufficient model interaction. Furthermore, individual interactions within a group vary significantly over time and last for extended periods. Existing video SSL methods, while achieving spatiotemporal consistency of representation pairs across large time slots, can only capture limited shared information. Predictive coding schemes, by considering temporal evolution through predicting future changes into a consistent space, maximize consistency between contrastive representation pairs and demonstrate the ability to learn representations of group behavior. However, these methods are limited by their inability to describe complex relationship transformations, significantly impacting the understanding of group behavior. Moreover, due to the complex contextual dynamics in group behavior, commonly used methods based on recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) have limited ability to model complex individual relationship transformations, significantly affecting performance. Therefore, the representation learning capabilities of existing video self-supervised learning (SSL) methods in this field still have considerable room for improvement. Summary of the Invention

[0005] The purpose of this application is to provide a self-supervised group behavior recognition method based on context-aware relation prediction coding. Based on context-aware relation prediction coding, a novel end-to-end self-supervised group feature model is proposed, which can capture contextual relations, perform self-learning of group behavior representations, and achieve efficient and accurate group behavior recognition under limited labeling conditions.

[0006] To achieve the above objectives, the first aspect of this application provides a self-supervised group behavior recognition method based on context-aware relation prediction encoding, comprising:

[0007] Acquire video data of group behavior;

[0008] A network structure for a self-supervised group feature model is constructed based on context-aware relation prediction encoding. The network structure includes a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder. The group label generator generates group labels for initializing individual representations based on the group behavior video data. The attention mechanism encoder generates high-level semantics for constructing group relationships based on the group labels. The hybrid context Transformer decoder predicts spatial and temporal group behavior features in future frames based on the high-level semantics and global scene context information.

[0009] The network structure is subjected to self-supervised training to obtain a feature self-learning target network;

[0010] The target network is obtained by fine-tuning the feature self-learning target network based on a small number of samples, and then the target network is used for self-supervised group behavior recognition.

[0011] In one implementation, the group marker generator is specifically used for:

[0012] The pre-trained I3D network is used as the backbone to extract feature maps for feature extraction from the input video segments;

[0013] Based on the results of feature extraction, an appearance feature branch and a pose cue branch are established respectively;

[0014] By connecting the appearance feature branch and the pose cue branch through a fully connected layer, a D-dimensional feature vector of each individual in the video clip is obtained;

[0015] The group label is obtained by combining the D-dimensional feature vectors.

[0016] In one implementation, the attention mechanism encoder is specifically used for:

[0017] The spatial and temporal interaction relationships between groups are explored, and high-level semantic encoding information of group behavior is obtained through a multi-head attention module, which serves as the high-level semantics.

[0018] In one implementation, the hybrid context Transformer decoder is specifically used for:

[0019] The scene context information of future video frames is obtained based on the last convolutional layer of the I3D network;

[0020] The high-level semantic encoding information of the group behavior and the scene context information are combined by the Transformer self-attention mechanism. Through convolution, normalization and pooling operations, the spatial and temporal characteristics of the group behavior in each future video frame are predicted step by step under the guidance of the scene context information of the future video frame.

[0021] In one implementation, the self-supervised training of the network structure includes:

[0022] During the prediction process of the hybrid context Transformer decoder, a joint loss is calculated based on the high-level semantic coding information of the group behavior, the scene context information, and the group behavior features.

[0023] Based on the joint loss, the parameters of the network structure are iteratively adjusted using the loss gradient backpropagation algorithm until training converges, and the feature self-learning target network is obtained.

[0024] In one implementation, the joint loss includes:

[0025] The inter-individual contrast loss and adversarial loss are used to perform fine-grained predictions and improve the prediction accuracy of group behavior characteristics at the individual level; the adversarial loss is used to improve the consistency of predictions.

[0026] In one implementation, fine-tuning the self-learning target network for group behavior features based on a small number of samples includes:

[0027] Based on a preset group behavior recognition task, the self-learning target network for group behavior features is fine-tuned using a small number of supervised learning samples to evaluate the self-learning representation ability of the self-learning target network for group behavior features.

[0028] Wherein, when the group behavior recognition task is to fine-tune the network structure for group behavior recognition, the fine-tuning of the feature self-learning target network based on a small number of samples includes: adding a linear classification layer after the feature self-learning target network, and fine-tuning the parameters of the feature self-learning target network and the linear classification layer under the constraints of labeled training samples.

[0029] A second aspect of this application provides a self-supervised group behavior recognition device based on context-aware relation prediction encoding, comprising:

[0030] The acquisition module is used to acquire video data of group behavior.

[0031] A construction module is used to build a network structure for a self-supervised model of group features based on context-aware relation prediction encoding. The network structure includes a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder. The group label generator generates group labels for initializing individual representations based on the group behavior video data. The attention mechanism encoder generates high-level semantics for constructing group relationships based on the group labels. The hybrid context Transformer decoder predicts spatial and temporal group behavior features of future frames based on the high-level semantics and global scene context information.

[0032] The training module is used to perform self-supervised training on the network structure to obtain a feature self-learning target network;

[0033] The fine-tuning module is used to fine-tune the feature self-learning target network based on a small number of samples to obtain the target network, and to perform self-supervised group behavior recognition through the target network.

[0034] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the first aspect or any embodiment of the first aspect.

[0035] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the first aspect or any embodiment of the first aspect.

[0036] As can be seen from the above, this application provides a self-supervised group behavior recognition method based on context-aware relation prediction coding, including acquiring group behavior video data; constructing a network structure of a self-supervised group feature model based on context-aware relation prediction coding, wherein the network structure includes a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder; performing self-supervised training on the network structure to obtain a feature self-learning target network; and fine-tuning the feature self-learning target network based on a small number of samples to obtain a target network for realizing group behavior recognition. This application provides a novel end-to-end framework, called a group behavior feature self-learning model based on context-aware relation prediction coding, for self-learning of group behavior representations. By capturing contextual relationships through context-aware relation prediction coding, high-level semantic coding information of group behavior and the scene context information can be combined to obtain a more comprehensive self-supervised group behavior representation. In addition, the method provided in this application has a higher recognition accuracy than existing methods, and has effectiveness and generalization, achieving efficient and accurate group behavior recognition under limited supervision. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A schematic diagram illustrating the process of constructing a network structure for a self-supervised model of group features, provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram illustrating a process for fine-tuning the feature self-learning target network, as provided in an embodiment of this application.

[0040] Figure 3 This is a diagram illustrating the effect of self-supervised group behavior recognition through a target network, provided as an embodiment of this application. Detailed Implementation

[0041] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0042] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0043] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0046] Example 1

[0047] This application provides a self-supervised group behavior recognition method based on context-aware relation prediction coding, such as... Figure 1 and Figure 2 As shown, the method includes:

[0048] Acquire video data of group behavior;

[0049] A network structure for a self-supervised group feature model is constructed based on context-aware relation prediction encoding. The network structure includes a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder. The group label generator generates group labels for initializing individual representations based on the group behavior video data. The attention mechanism encoder generates high-level semantics for constructing group relationships based on the group labels. The hybrid context Transformer decoder predicts spatial and temporal group behavior features in future frames based on the high-level semantics and global scene context information.

[0050] The network structure is subjected to self-supervised training to obtain a feature self-learning target network;

[0051] The feature self-learning target network is fine-tuned based on a small number of samples to obtain the target network, which is then used for self-supervised group behavior recognition.

[0052] In one implementation, the aforementioned group behavior video data includes data from two group behavior datasets: VolleyTactic Dataset and Choi'sNew Dataset. The aforementioned group behavior video data is preprocessed, with all video frames scaled to a resolution of 720×1280. T=10 frames are selected as the input for training the self-supervised model of group features in subsequent steps, so as to realize the analysis of group behavior on multiple video datasets.

[0053] Optionally, the group marker generator is specifically used for:

[0054] The pre-trained I3D network is used as the backbone to extract feature maps for feature extraction from the input video segments;

[0055] Based on the results of feature extraction, an appearance feature branch and a pose cue branch are established respectively;

[0056] By connecting the appearance feature branch and the pose cue branch through a fully connected layer, a D-dimensional feature vector of each individual in the video clip is obtained;

[0057] The group label is obtained by combining the D-dimensional feature vectors.

[0058] In one implementation, the group tag generator is a preprocessing module containing rich individual information from the video to initialize individual representations. Specifically, given an input video segment X of T frames... video Since the position and movement of the body's joints determine an individual's actions, the first step is to extract feature maps for X using a pre-trained I3D network as the backbone. videoFeature extraction is performed; the swarm tag generator establishes two branches using appearance features and pose cues, which are further connected through fully connected layers to form D-dimensional feature vectors for each individual. Finally, these individual D-dimensional feature vectors are combined to form the final swarm tag X. G .

[0059] Optionally, the attention mechanism encoder is specifically used for:

[0060] The spatial and temporal interaction relationships between groups are explored, and high-level semantic encoding information of group behavior is obtained through a multi-head attention module, which serves as the high-level semantics.

[0061] In one implementation, the purpose of the attention mechanism encoder is to summarize observation X. video The algorithm fully explores the group relationships and interactions between groups. Since group behavior involves complex inter-individual interactions, the attention mechanism encoder uses a multi-head attention module to extract features from individual information and complex inter-individual interactions in the group tags, obtaining high-level semantic encoding information of group behavior, which serves as the high-level semantics. Alternatively, in other embodiments, the attention mechanism encoder can also obtain the high-level semantics in other ways, which are not limited here.

[0062] Optionally, the hybrid context Transformer decoder is specifically used for:

[0063] The scene context information of future video frames is obtained based on the last convolutional layer of the I3D network;

[0064] The high-level semantic encoding information of the group behavior and the scene context information are combined by the Transformer self-attention mechanism. Through convolution, normalization and pooling operations, the spatial and temporal characteristics of the group behavior in each future video frame are predicted step by step under the guidance of the scene context information of the future video frame.

[0065] In one implementation, the hybrid context Transformer decoder acquires observed relational semantics based on the high-level semantics of the attention mechanism encoder, and progressively extracts spatial and temporal group behavior features for each future video frame under the guidance of scene context information from future video frames. Specifically, the hybrid context Transformer decoder is based on a Transformer architecture, where relational semantics and scene context information are combined with a Transformer self-attention mechanism. The scene of the future video frame T' is extracted from the last convolutional layer of the I3D model and can be represented as X. s Through convolution, normalization, and pooling operations, the final scene context information representation X is generated. SenceThe pooled relational semantic features of frame t generated from the attention mechanism encoder are treated as query Q, while the scene context is treated as X. Sence As keys K and V, in each prediction step, relational semantic cues capture future relevant scene content from K by calculating the similarity between Q and K. The attention operation Con_Att produces a weighted sum of V as the output, where Q is further added to update the self-attention. The prediction result for frame t+1 can be represented as... This serves as the output of the Hybrid Context Transformer decoder. The Hybrid Context Transformer decoder also includes an Add-Norm layer and a feedforward network layer. The group feature self-supervised model treats the embeddings of the previous T frames as a sliding window input, and the prediction results are used in the next step of encoding and decoding to predict the next frame.

[0066] Optionally, the self-supervised training of the network structure includes:

[0067] During the prediction process of the hybrid context Transformer decoder, a joint loss is calculated based on the high-level semantic coding information of the group behavior, the scene context information, and the group behavior features.

[0068] Based on the joint loss, the parameters of the network structure are iteratively adjusted using the loss gradient backpropagation algorithm until training converges, and the feature self-learning target network is obtained.

[0069] Optionally, the joint loss includes:

[0070] The inter-individual contrast loss and adversarial loss are used to perform fine-grained predictions and improve the prediction accuracy of group behavior characteristics at the individual level; the adversarial loss is used to improve the consistency of predictions.

[0071] In one implementation, since group behavior involves complex individual relationships across both spatial and temporal transitions, attention is paid to individual contextual information, and an inter-individual contrastive loss is designed for fine-grained prediction. The inter-individual contrastive loss function is:

[0072]

[0073] Here, i is the individual index, and j represents other indices, including different spatial indices or batch indices. In this function, only the predicted and true representations of individuals spatially aligned at the same time are considered positive sample pairs; others are negative sample pairs. The inter-individual contrastive loss ensures that the self-supervised model of the group features can predict individual dependencies at a fine-grained local frame level.

[0074] Furthermore, the adversarial loss is used to improve prediction consistency. This involves incorporating the predicted group features of future T' frames from the hybrid context Transformer decoder. Feature descriptors extracted from T-frame video, treated as fake samples These are considered real samples. Based on these real / fake sample pairs, the generator G and discriminator D engage in adversarial competition, enabling the framework to generate sequences that are difficult to distinguish from real samples, thereby improving prediction consistency. This can be described as follows:

[0075]

[0076] The distribution E(·) lies on the training sequence. The discriminator D is a classic Transformer structure followed by pooling and linear classification layers, outputting the true probability.

[0077] Furthermore, the sum of the two losses mentioned above constitutes the joint loss, expressed as:

[0078]

[0079] Optionally, since the representation learning ability of the self-supervised model for group features cannot be directly evaluated, this embodiment of the application uses the feature self-learning target network obtained through the above self-supervised training to fine-tune the downstream recognition task based on supervised learning, and then evaluates it, such as... Figure 2 As shown, the fine-tuning of the feature self-learning target network based on a small number of samples includes:

[0080] Based on a preset group behavior recognition task, the feature self-learning target network is fine-tuned using a small number of supervised learning samples to evaluate the representation self-learning ability of the feature self-learning target network.

[0081] Wherein, when the group behavior recognition task is to fine-tune the network structure for group behavior recognition, the fine-tuning of the feature self-learning target network based on a small number of samples includes: adding a linear classification layer after the feature self-learning target network, and fine-tuning the parameters of the feature self-learning target network and the linear classification layer under the constraints of labeled training samples.

[0082] In one implementation, in supervised learning for group behavior recognition, the input group behavior video samples follow the same sampling procedure as the self-supervised model for group features described above. In this case, during the training phase, all frames in the video samples can be used for training, without needing to consider prediction. Furthermore, when group behavior recognition is evaluated using a preset group behavior recognition task, cross-entropy loss can be used to train the classifier. Specifically, in training a linear classifier for group behavior recognition, all samples in the training set are input into the network to train the classifier; in fine-tuning the entire network structure for group behavior recognition, 10% of the labeled samples are randomly selected to fine-tune all parameters in the network structure. During the testing phase, a standard procedure is followed, obtaining the same sequence length from the video samples as in the training phase for testing, thereby enabling the analysis of group behavior on the video dataset.

[0083] As can be seen from the above, this application provides a method for constructing a self-supervised model of group features, including acquiring group behavior video data; constructing a network structure of the self-supervised model of group features based on context-aware relation prediction encoding, wherein the network structure includes a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder; performing self-supervised training on the network structure to obtain a feature self-learning target network; and fine-tuning the feature self-learning target network based on a small amount of supervised learning to obtain a target network for realizing group behavior recognition. This application provides a novel end-to-end framework, called a group behavior feature self-learning model based on context-aware relation prediction encoding, for self-learning of group behavior representations. By capturing contextual relationships through context-aware relation prediction encoding, the high-level semantic encoding information of group behavior and the scene context information can be combined to obtain a more comprehensive self-supervised group behavior representation.

[0084] Example 2

[0085] The following experiment demonstrates the effectiveness of the self-supervised group behavior recognition method provided in Embodiment 1 of this application.

[0086] Based on the self-supervised group behavior recognition method proposed in this application, experiments were conducted on two widely used group behavior datasets (VolleyTactic Dataset and Choi's New Dataset), and comparisons were made with state-of-the-art methods on all datasets. Experimental results show that the self-supervised group behavior recognition method proposed in this application outperforms existing methods in terms of recognition accuracy, confirming the effectiveness and generalization of the method. The effect of the above-mentioned group behavior recognition method based on a context-aware relation prediction encoding self-learning model of group behavior features is illustrated in the figure below. Figure 3As shown, the real value is the group behavior label of the video, and the recognition result is the group behavior description obtained based on the group behavior recognition method. It is clear from the figure that the group behavior recognition method proposed in this application has superiority in self-supervised group behavior representation learning.

[0087] Example 3

[0088] This application provides a self-supervised group behavior recognition device based on context-aware relation prediction coding, including an acquisition module, a construction module, a training module, and a fine-tuning module.

[0089] The acquisition module is used to acquire video data of group behavior.

[0090] The construction module is used to build a network structure for a self-supervised model of group features based on context-aware relation prediction encoding. The network structure includes a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder. The group label generator generates group labels for initializing individual representations based on the group behavior video data. The attention mechanism encoder generates high-level semantics for constructing group relationships based on the group labels. The hybrid context Transformer decoder predicts spatial and temporal group behavior features of future frames based on the high-level semantics and global scene context information.

[0091] The training module is used to perform self-supervised training on the network structure to obtain a feature self-learning target network;

[0092] The fine-tuning module is used to fine-tune the feature self-learning target network based on a small number of samples to obtain the target network, and then use the target network to perform self-supervised group behavior recognition.

[0093] As can be seen from the above, the embodiments of this application provide a self-supervised group behavior recognition device based on context-aware relation prediction coding. By capturing contextual relationships through context-aware relation prediction coding, the high-level semantic coding information of group behavior and the scene context information can be combined to obtain a more comprehensive self-supervised group behavior representation. Furthermore, the device provided in the embodiments of this application has a higher recognition accuracy than existing methods, demonstrating effectiveness and generalization, and achieving efficient and accurate group behavior recognition under limited supervision.

[0094] Example 4

[0095] This application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any of the steps in Embodiment 1 by running the computer program stored in the memory.

[0096] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0097] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0098] As can be seen from the above, the electronic device provided in this application embodiment can implement the self-supervised group behavior recognition method as described in Embodiment 1 by running a computer program, resulting in a new end-to-end framework called a group feature self-supervised model, used for group behavior representation learning. By capturing contextual relationships through context-aware relation prediction encoding, it not only considers spatial interaction contextual relationships but also overall scene temporal change contextual relationships, thereby obtaining a more comprehensive representation of group behavior features.

[0099] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0100] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0102] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments, and can be referred to each other, without further elaboration.

[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of the modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0105] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A self-supervised group behavior recognition method based on context-aware relationship prediction coding, characterized in that, The method comprises the following steps: acquiring group behavior video data; constructing a network structure of a group feature self-supervision model based on context-aware relationship prediction coding, the network structure comprising a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder, wherein the group label generator generates a group label for initializing individual representation according to the group behavior video data; the attention mechanism encoder generates high-level semantics for constructing group relationships according to the group label; and the hybrid context Transformer decoder predicts spatial and temporal group behavior features of future frames according to the high-level semantics and global scene context information; performing self-supervised training on the network structure to obtain a feature self-learning target network; fine-tuning the feature self-learning target network based on a small amount of samples to obtain a target network, and performing self-supervised group behavior recognition through the target network; the group label generator is specifically configured to: extract features from an input video segment by taking a pre-trained I3D network as a backbone to extract feature maps; establish an appearance feature branch and a pose clue branch according to the feature extraction results, respectively; connect the appearance feature branch and the pose clue branch through a fully connected layer to obtain a D-dimensional feature vector of each individual in the video segment; combine the D-dimensional feature vectors to obtain the group label; the hybrid context Transformer decoder is specifically configured to: obtain scene context information of future video frames according to the last convolutional layer of the I3D network; combine the group behavior high-level semantic coding information and the scene context information through a Transformer self-attention mechanism, and gradually predict the spatial and temporal group behavior features of each future frame under the guidance of the scene context information of the future video frames through convolution, normalization, and pooling operations; the self-supervised training of the network structure comprises: during the prediction process of the hybrid context Transformer decoder, calculating a joint loss based on the group behavior high-level semantic coding information, the scene context information, and the group behavior features; based on the joint loss, iteratively adjusting parameters of the network structure through a loss gradient backpropagation algorithm until training converges, and obtaining the feature self-learning target network.

2. The self-supervised group behavior recognition method of claim 1, wherein, the attention mechanism encoder is specifically configured to: mine spatial and temporal interaction relationships among groups to obtain group behavior high-level semantic coding information through a multi-head attention module as the high-level semantics.

3. The self-supervised group behavior recognition method of claim 1, wherein, the joint loss comprises: an individual-to-individual contrast loss and an adversarial loss, wherein the individual-to-individual contrast loss is used for fine-grained prediction to improve the prediction accuracy of individual-level group behavior features; and the adversarial loss is used to improve the consistency of prediction.

4. The self-supervised group behavior recognition method of claim 2, wherein, the fine-tuning of the feature self-learning target network based on a small amount of samples comprises: based on a small amount of supervised learning samples, fine-tuning the feature self-learning target network according to a preset group behavior recognition task to evaluate the representation self-learning ability of the feature self-learning target network. In the group behavior recognition task, the fine-tuning of the entire network structure for group behavior recognition comprises: adding a linear classification layer to the feature self-learning target network, and fine-tuning each parameter of the feature self-learning target network and the linear classification layer under the constraint of the labeled training sample.

5. A self-supervised group behavior recognition apparatus based on context-aware relationship prediction coding, characterized by, The method comprises the following steps: An acquisition module is configured to acquire group behavior video data. A construction module is configured to construct a network structure of a group feature self-supervision model based on context-aware relationship prediction coding, wherein the network structure comprises a group label generator, an attention mechanism encoder, and a hybrid context Transformer decoder. The group label generator is configured to generate a group label for initializing individual representation according to the group behavior video data. The attention mechanism encoder is configured to generate high-level semantics for constructing group relationship according to the group label. The hybrid context Transformer decoder is configured to predict spatial and temporal group behavior features of future frames according to the high-level semantics and global scene context information. The group label generator is specifically configured to: extract feature maps from a pre-trained I3D network as a backbone for feature extraction of an input video segment; establish an appearance feature branch and a pose clue branch based on the results of feature extraction, respectively; connect the appearance feature branch and the pose clue branch through a fully connected layer to obtain a D-dimensional feature vector of each individual in the video segment; combine the D-dimensional feature vectors to obtain the group label. The hybrid context Transformer decoder is specifically configured to: obtain scene context information of future video frames from the last convolutional layer of the I3D network; combine the group behavior high-level semantic coding information and the scene context information through a Transformer self-attention mechanism, and gradually predict the spatial and temporal group behavior features of each future frame under the guidance of the scene context information of the future video frames through convolution, normalization, and pooling operations. A training module is configured to perform self-supervised training on the network structure to obtain a feature self-learning target network.

6. An electronic device comprising: The self-supervised training on the network structure comprises: calculating a joint loss based on the group behavior high-level semantic coding information, the scene context information, and the group behavior features during the prediction process of the hybrid context Transformer decoder; iteratively adjusting each parameter of the network structure through a loss gradient backpropagation algorithm based on the joint loss until the training converges and the feature self-learning target network is obtained. A fine-tuning module is configured to fine-tune the feature self-learning target network based on a small number of samples to obtain a target network, and perform self-supervised group behavior recognition through the target network. A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 4.

Images