Method and system for discovering social interaction groups in a video based on environmental perception networks

Abstract

The application provides a method and system for discovering social interaction groups in a video based on environmental perception network, comprising: environmental perception network, designing two types of environmental perception visual cues to represent factors affecting social interaction; scene perception visual cues, using a 3D network to extract input video segment features as visual representation, and using an encoder to extract space-time features; human perception visual cues, using a bounding box and a region of interest as visual representation of openness, orientation and position of posture affecting interaction in psychology; extracting distance features from the bounding box and personal features from the region of interest, the distance features and the personal features as visual representation of human environment; fusing the space-time features, the distance features and the personal features to obtain high-order social interaction intensity features, and obtaining an interaction intensity matrix between people. The application uses psychology to discover social interaction from a more essential angle, and quantifies the interaction intensity between people by using a deep learning method.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method and system for discovering social interaction groups in videos based on an environment-aware network. Background Technology

[0002] Social interaction refers to behaviors that attempt to influence or consider the subjective experiences or intentions of others. Social interaction is very common in human societies and has played a significant role in human evolution. Groups formed through social interaction are called social interaction groups. Discovering social interaction groups is a fundamental task in the analysis of human social behavior, and it can be well extended to some group-related computer vision tasks, such as group anomaly detection and group action recognition.

[0003] Current methods for social interaction recognition include:

[0004] (1) Discovering social interactions based on the proximity between people. One type of this method uses the F-formation proposed in sociology, which uses a predefined shape to find people in the video who match the shape, thus discovering social interactions. However, this type of method requires additional information, such as the orientation of people's heads, which is difficult to capture directly from the video.

[0005] (2) Discovering social interactions based on predefined action categories. The characteristic of this type of method is that it discovers social interactions between people in a video based on predefined interaction types. However, the biggest drawback of this type of method is that it requires specific action categories, while our actual social interactions are ever-changing, and providing specific action categories is insufficient to cover all types of social interactions.

[0006] The applicant's previous invention application, CN114495159A, proposed a social interaction group detection method using a self-attention mechanism. This method can quantify the strength of interactions between people in an image and identify social interaction groups within the image, based solely on image information. However, this patent only detects social interactions from an image perspective and is not well-suited for detecting social interactions in video information with temporal information. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this application is to provide a method and system for discovering social interaction groups in videos based on environmental awareness networks.

[0008] One aspect of this application provides a method for discovering social interaction groups in videos based on an environment-aware network, comprising:

[0009] An environment perception network is provided, which designs two types of environmental visual cues to represent factors influencing social interaction: scene-aware visual cues and human-aware visual cues. Each type of visual cue consists of a visual representation and a feature extraction network, wherein:

[0010] The scene-aware visual cues are obtained by using a 3D network to extract features from the input video segment as a visual representation, and by using an encoder to extract spatiotemporal features.

[0011] The aforementioned human-perceived visual cues utilize bounding boxes and regions of interest as visual representations of posture openness, orientation, and position that influence interaction in psychology. Specifically, a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the regions of interest. The extracted distance features and personal features serve as a visual representation of the human environment.

[0012] An environment perception mechanism based on Transformer is used as a feature fusion network to fuse the spatiotemporal features, distance features and personal features to obtain high-order social interaction intensity features.

[0013] The dimensionality of the social interaction intensity features is reduced to obtain the interaction intensity matrix between people, thereby discovering social interaction groups in the video.

[0014] This application uses scene visual cues with temporal characteristics to complete social interaction discovery, and adopts a transformer-based environment perception mechanism as a feature fusion network to complete feature fusion from different visual cues, which can adaptively capture social interaction information between people from different angles.

[0015] Optionally, the scene-aware visual cues correspond to the scene environment, which is considered from a global perspective, including static targets and dynamic information around a person; the human-aware visual cues correspond to the human environment, which is considered from a local perspective, including the appearance and location information of each person and other people around them.

[0016] Optionally, the step of using a 3D network to extract features from the input video segment as a visual representation and using an encoder to extract spatiotemporal features includes: extracting input video features X using an I3D network pre-trained on Kinetics-400. I As a visual representation of the scene, spatiotemporal features ST are extracted using an encoder E1 consisting of convolutional layers, pooling layers, and temporal averaging operations, and are used as scene features.

[0017] Optionally, the use of bounding boxes and regions of interest as visual representations of pose openness, orientation, and position that influence interaction in psychology, wherein:

[0018] By using each person's bounding box as a visual representation of distance information that influences interaction, a distance feature extractor consisting of relative position encoding and a neural network is used to obtain the distance feature d′ between any two people. ij ,

[0019] Using each person's region of interest as a visual representation of the openness and orientation of posture that influences interaction, features are extracted from these regions using a personal feature extractor composed of a spatiotemporal encoder to obtain each person's individual features h. i .

[0020] Optionally, the step of fusing the spatiotemporal features, distance features, and personal features using a Transformer-based environment perception mechanism to obtain higher-order social interaction intensity features includes:

[0021] Calculating human perception and interaction relationships g ij Using the distance feature d′ ij As the initial g ij ;

[0022] Using graph-guided updates, the personal feature h i renew;

[0023] Using a Transformer encoder, combined with the spatiotemporal features ST, the personal features h are processed. i renew;

[0024] Perform a negative exponential operation on the updated individual characteristics to obtain paired relationships P. ij ;

[0025] The final human-perceived interaction relationship g is obtained using a moving average method. ij ;

[0026] The social interaction intensity feature I is obtained by calculating the outer product of the human perceived interaction relationship and the spatiotemporal features.

[0027] Optionally, the dimensionality reduction of the social interaction intensity features to obtain the interpersonal interaction intensity matrix includes:

[0028] The social interaction intensity features are reduced in dimensionality using affine transformation and symmetry operations, and are then transformed into an interaction intensity matrix R.

[0029] Optionally, the step of using affine transformation and symmetry operations to reduce the dimensionality of the social interaction intensity features, transforming the social interaction intensity features into an interaction intensity matrix R, includes:

[0030] We reduce the dimensionality of high-dimensional social interaction strength features and use a softmax operation to constrain them to the range of 0-1, i.e.:

[0031] X = Softmax(W s T S+m s ), where W s and m s All are learnable parameters, and X is the probability pattern matrix;

[0032] To maintain the symmetry of the interaction strength matrix, a symmetry operation is used, setting its diagonal elements to 0, thus obtaining the interaction strength matrix R, i.e.:

[0033] Where ⊙ represents the Hadamard product, and U is an N×N matrix with 0 elements on the diagonal and 1 elements on the rest.

[0034] Optionally, the method further includes:

[0035] After obtaining the interaction intensity matrix, a pairwise interaction constraint loss function is established for the interaction intensity matrix between each pair of people in the video, and the interaction critical loss L is used. thr To increase differentiation, use smoothing F β Loss function L β This is to alleviate the inaccurate classification caused by category imbalance.

[0036] Optionally, a pairwise interaction constraint loss function is established for the interaction strength matrix between each pair of people in the video, utilizing the interaction critical loss L. thr To increase differentiation, use smoothing F β Loss function L β To mitigate classification inaccuracies caused by class imbalance, the following measures are taken:

[0037] The hyperbolic tangent function is used to perform thresholding on the interaction intensity matrix, i.e. Where θ is the threshold and c is the hyperparameter, then using the given labels in the dataset (i.e., people belonging to the same group are assigned the same number), the true interaction matrix G is obtained, and the loss function is calculated as follows:

[0038] L thr =-∑ i,j [G ij log(C ij )+(1-C ij log(1-C) ij )], where C ij This indicates whether there is interaction between any two people. People belonging to the same group have interaction, which is set to 1; no interaction is set to 0. C ij This represents the interaction strength matrix between any two people after processing with the hyperbolic tangent function, ranging from 0 to 1;

[0039] Smooth Fβ loss function for:

[0040]

[0041] Where tp=∑C⊙G, fp=∑C⊙(1-G), fn=∑(1-C)⊙G;

[0042] The final pairwise interaction constraint loss function is obtained by linearly combining the two loss functions: Where λ is a hyperparameter.

[0043] A second aspect of this application provides a system for discovering social interaction groups in videos based on an environment-aware network, comprising:

[0044] An environmental perception network is designed with two types of environmental perception visual cues to represent factors influencing social interaction, wherein the two types of environmental perception visual cues include scene perception visual cues and human perception visual cues.

[0045] Scene-aware visual cue module: It uses a 3D network to extract features from the input video segment as a visual representation, and uses an encoder to extract spatiotemporal features;

[0046] The Human Perception Visual Cue Module utilizes bounding boxes and regions of interest (ROIs) as visual representations of pose openness, orientation, and position that influence interaction in psychology. Specifically, a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the ROIs. The extracted distance and personal features serve as a visual representation of the human environment.

[0047] Environmental perception mechanism module: The environment perception mechanism based on Transformer is used to fuse the spatiotemporal features of the scene perception visual cue extraction module, the distance features of the human perception visual cue module, and personal features to obtain high-order social interaction intensity features.

[0048] Interaction discovery module: Reduces the dimensionality of the social interaction intensity features to obtain the interaction intensity matrix between people, thereby discovering social interaction groups in the video.

[0049] Compared with the prior art, this application has at least one of the following beneficial effects:

[0050] The method and system for discovering social interaction groups in videos based on environmental perception networks provided in this application borrows from the feedback relationship between environment and human behavior in psychology. Under this guidance, a psychology-guided environmental perception network is designed. It designs two types of environmental perception visual cues to represent factors that influence social interaction and extracts corresponding visual features, thereby realizing the measurement of the intensity of social interaction between people.

[0051] The method and system for discovering social interaction groups in videos based on environmental perception networks provided in this application, based on the spatiotemporal features, distance features, and personal features obtained from two types of environmental perception visual cues, designs a Transformer-based environmental perception mechanism to integrate the features output by the two types of environments, explore the interaction relationships between people, and construct a high-order social interaction intensity feature that adapts to complex interaction situations. This enables the discovery of social interactions from a more fundamental perspective, and also realizes the quantification of the interaction intensity between people using deep learning methods.

[0052] The method and system for discovering social interaction groups in videos based on environmental perception networks provided in this application construct a pairwise interaction constraint loss function, and further utilize the interaction critical loss function L... thr and smooth F β Loss function L β This can improve the discriminative power of the predicted interaction intensity matrix and alleviate the class imbalance problem. Attached Figure Description

[0053] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0054] Figure 1 This is a flowchart illustrating a method for discovering social interaction groups in a video based on an environment-aware network, according to an embodiment of this application.

[0055] Figure 2 This is a schematic diagram showing the specific parameters and composition of the environment perception network structure in a preferred embodiment of this application. Detailed Implementation

[0056] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0057] This application provides a method to discover social interactions from a more fundamental perspective. It involves constructing a computer vision model, specifically an environmental perception network, based on psychological theories, to measure the strength of social interactions between people in videos and identify social interaction groups. In behavioral psychology, human behavior and the environment constitute a feedback loop, and social interaction is a type of behavior; therefore, social interactions between people and the environment are mutually influential. Based on this, this application will perform computer vision modeling of the environment surrounding people. Specifically, the environment around people is divided into two parts: one part designs scene-aware visual cues, and the other part designs human-aware visual cues to represent the human environment of those around them.

[0058] Reference Figure 1 The diagram shown is a flowchart of a method for discovering social interaction groups in a video based on an environmental awareness network, according to an embodiment of this application. Specifically, this embodiment can be implemented according to the following steps:

[0059] S100, providing an environment perception network, wherein the environment perception network is designed with two types of environmental perception visual cues to represent factors that influence social interaction;

[0060] Scene-aware visual cues are used to represent the scene environment, considering static objects and dynamic information around a person from a global perspective. Human-aware visual cues are used to represent the human environment, considering the appearance and location information of others around each person from a local perspective. Notably, each type of visual cue consists of a visual representation and a feature extraction network. The feature extraction network can be selected as needed.

[0061] S200, a scene-aware visual cue, uses a 3D network to extract features from the input video segment as a visual representation, and uses an encoder to extract spatiotemporal features;

[0062] S300, a human-perceived visual cue, utilizes bounding boxes and regions of interest as visual representations of posture openness, orientation, and position that influence interaction in psychology. Specifically, a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the regions of interest. The extracted distance features and personal features serve as a visual representation of the human environment.

[0063] In this step, personal feature extraction can be achieved using a space-time encoder. For example, a space-time encoder can consist of two parts: a spatial behavior encoder based on a vision transformer encoder and a temporal averaging encoder. Of course, in other embodiments, other designs of space-time encoders can also be used, as long as they can perform the feature extraction described above.

[0064] S400 employs a Transformer-based environment perception mechanism as a feature fusion network to fuse the spatiotemporal features, distance features, and personal features to obtain high-order social interaction intensity features.

[0065] S500, the dimensionality of the social interaction intensity features is reduced to obtain the interaction intensity matrix between people, thereby discovering social interaction groups in the video.

[0066] In this embodiment, two types of environmental visual cues are designed, and human-perceived visual cues are divided into two modules. One module uses bounding boxes as a positional representation of distance relationships influencing social interactions between people, and extracts distance features using relative position encoding combined with a neural network. The other module uses regions of interest as visual representations of pose openness and orientation influencing social interactions between people, and extracts personal features through a spatiotemporal encoder. The spatiotemporal features, distance features, and personal features are fed into a Transformer-based environmental perception mechanism, which acts as a feature fusion network to construct high-order social interaction intensity features to adapt to different situations of social interaction. This achieves the discovery of social interactions from a fundamental psychological perspective and quantifies the intensity of social interactions between people.

[0067] In a preferred embodiment, refer to Figure 2 As shown, the method for discovering social interaction groups in videos based on environmentally perceptive networks includes the following specific steps:

[0068] Step S1: Divide the environment into two parts: the scene environment, which reflects the overall visual information of the person's surroundings, including static targets and motion, and the human environment, which reflects the visual information of people around the person other than oneself.

[0069] Step S2: Environmental perception feature extraction.

[0070] Extraction methods will be introduced for the two main types of environments mentioned in S1.

[0071] In this embodiment, for the scene environment: a 3D network is used to extract features from the input video segment as a visual representation, and an encoder is used to extract spatiotemporal features; specifically, scene-aware visual cues are designed, and the input video features X are extracted using an I3D network pre-trained on Kinetics-400. I As a visual representation of the scene, spatiotemporal features ST are extracted using an encoder E1 consisting of convolutional layers, pooling layers, and temporal averaging operations, which serve as scene features.

[0072] In this embodiment, for the human environment: human perception of visual cues utilizes bounding boxes and regions of interest as visual representations of posture openness, orientation, and position that influence interaction in psychology; wherein, a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the regions of interest, and the extracted distance features and personal features serve as the visual representation of the human environment.

[0073] Specifically, by using each person's bounding box as a visual representation of distance information that influences interaction, a distance feature extractor consisting of relative position encoding and a neural network is used to obtain the distance feature d′ between any two people. ij ;

[0074] Based on the bounding box coordinates (x, y, w, h, corresponding to the x-coordinate of the center of the bounding box, the y-coordinate of the center of the bounding box, the width of the bounding box, and the height of the bounding box, respectively) of any two people i and j in the video: Calculate the relative distance B between them. ij :

[0075]

[0076] B ij Encode, given encoding frequency Where d model Let k represent the dimension, k represent the offset, and K represent the maximum offset. The relative position information is obtained by performing a tensor product of the relative position distance and frequency. here Represent the tensor product. Let E... ij Converting to a vector and calculating using sine and cosine, the relative position code is obtained as: D ij =[sin(E ij ),cos(E ij Using an FC1 neural network and activation function, the pairwise interaction distance relationship d is obtained. ij , that is, d ij =ReLU(FC1(D) ij Considering that the distance between people is inversely proportional to the intensity of interaction, the final distance feature is:

[0077] Specifically, by using each person's region of interest as a visual representation of the openness and orientation of posture that influences interaction, a personal feature extractor composed of a spatiotemporal encoder is designed to extract features from these regions, thereby obtaining each person's individual features h. i By designing a space-time encoder, features are extracted to obtain the individual characteristics h of each person. i The approach is as follows: For the region of interest of the i-th person in the q-th frame... Using the Vision Transformer encoder as a spatial behavior encoder (SBEncoder), we extract the personal features of the i-th person in the q-th frame. By designing a time encoder TE based on a neural network q Add to personal characteristics: By performing a time-series averaging operation, the personal characteristics of the i-th person in this video segment are obtained. Where Q is the total number of video frames in the video segment.

[0078] S3. The spatiotemporal features, distance features, and personal features are fused using a Transformer-based environmental perception mechanism to obtain high-order social interaction intensity features.

[0079] In this step, an environment-awareness mechanism was designed to obtain social interaction intensity features based on the interaction element features extracted in S2. The specific steps are as follows:

[0080] S31. Calculate human-perceived interaction relationships g ij , using d′ ij As the initial g ij .

[0081] S32. Use graphs to guide updates, focusing on individual characteristics h. i Update, that is:

[0082] S33. Using the Transformer encoder, combined with the spatiotemporal features ST, for h i The update involves using a neural network to connect ST and h. i We achieve feature dimension alignment, and then use the encoder framework in Transformer, combined with the updated ST, to update the individual features, i.e., h. i =TransformerEncoder(h i +Φ ST (ST)).

[0083] S34. Perform a negative exponential operation on the updated personal characteristics to obtain paired relationships P. ij The operation is as follows:

[0084] P ij =exp(-||h i -h j ||).

[0085] S35. Obtain the final human-perceived interaction relationship g using a moving average method. ij The operation is as follows:

[0086] gij =αg ij +(1-α)g ij .

[0087] S36. Calculate the outer product of human perceived interaction relationships and scene features to obtain the social interaction strength feature I. The method is as follows: Here I∈R N×N×dim N represents the number of people in the video, and LayerNorm represents layer normalization.

[0088] This embodiment integrates the features of two types of environmental outputs through the above-mentioned Transformer-based environmental perception mechanism, explores the interaction relationship between people, and constructs a high-order interaction intensity feature that adapts to complex interaction situations. It can discover social interaction from a more fundamental perspective, and also realizes the quantification of the interaction intensity between people using deep learning methods.

[0089] S4: Dimensionally reduce the social interaction intensity features obtained in S3 to obtain the interaction intensity matrix between people, thereby discovering social interaction groups in the video.

[0090] In this embodiment, the interaction strength features are converted into an interaction strength matrix R using affine transformation and symmetry operations, specifically:

[0091] First, the high-dimensional interaction intensity features are reduced in dimensionality, and then the softmax operation is used to constrain them to the range of 0-1. The method is: X = Softmax(W s T S+m s ), where W s and m s All of these are learnable parameters, and X is the probability pattern matrix.

[0092] To ensure the symmetry of the interaction intensity matrix, a symmetry operation is used. Here, since a person does not interact with themselves, the diagonal elements of the interaction intensity matrix are assigned the value 0, resulting in the interaction intensity matrix. This operation is as follows: Where ⊙ represents the Hadamard product, and U is an N×N matrix with 0 elements on the diagonal and 1 elements on the rest.

[0093] Furthermore, in this embodiment, based on the above, an optimization objective is further established to optimize the entire environmental perception network.

[0094] Specifically, a pairwise interaction constraint loss function is established for the interaction strength matrix between each pair of people in the video. It consists of an interaction critical loss function L that increases discriminative power. thr And smoothing F to mitigate classification inaccuracies caused by class imbalance β Loss function Lβ Composition. The two loss functions are as follows:

[0095] (1) To increase the discriminative power of the interaction intensity matrix and make it easily distinguishable by a threshold, a thresholded loss function is used. The method is as follows: first, the hyperbolic tangent function is used to threshold the interaction intensity matrix, i.e. Where θ is the threshold and c is the hyperparameter, then using the given labels in the dataset (i.e., people belonging to the same group are assigned the same number), the true interaction matrix G is obtained, and the loss function is calculated as: L thr =-∑ i,j [G ij log(C ij )+(1-G ij log(1-C) ij )], where G ij This indicates whether there is interaction between any two people. People belonging to the same group have interaction, which is set to 1; no interaction is set to 0. C ij This represents the interaction strength matrix between any two people after processing with the hyperbolic tangent function, ranging from 0 to 1.

[0096] (2) Considering that in a video segment scenario, the number of paired social interactions is much smaller than the number of non-paired interactions, which can lead to class imbalance, and that F1-score is a good indicator of class imbalance, it is modified as the loss function. Using the existing smoothed F1 loss function may cause true negative samples to be incorrectly predicted as positive samples, thus increasing the number of false positives; therefore, it is modified to a smoothed F1 loss function. β The loss function operates as follows:

[0097]

[0098] Where tp=∑C⊙G, fp=∑C⊙(1-G), fn=∑(1-C)⊙G.

[0099] The final pairwise interaction constraint loss function can be obtained by linearly combining the two loss functions: Where λ is a hyperparameter, which can be set to 10 in a specific embodiment.

[0100] The smooth F1 loss function described above is an existing technique; for details, please refer to Bénédict G, Koops V, Odijk D, et al. SigmoidF1: A smooth F1 score surrogate loss for multilabel classification[J]. arXiv preprint arXiv:2108.10566,2021.

[0101] The embodiments described above in this application borrow from psychological theories to discover social interactions from a more fundamental perspective, and also realize the quantification of the intensity of interactions between people using deep learning methods.

[0102] Based on the same technical concept, this application also provides a system for discovering social interaction groups in videos based on an environment-aware network, including:

[0103] An environmental perception network is designed with two types of environmental perception visual cues to represent factors influencing social interaction, wherein the two types of environmental perception visual cues include scene perception visual cues and human perception visual cues.

[0104] Scene-aware visual cue module: It uses a 3D network to extract features from the input video segment as a visual representation, and uses an encoder to extract spatiotemporal features;

[0105] The Human Perception Visual Cue Module utilizes bounding boxes and regions of interest (ROIs) as visual representations of pose openness, orientation, and position that influence interaction in psychology. Specifically, a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the ROIs. The extracted distance and personal features serve as a visual representation of the human environment.

[0106] Environmental perception mechanism module: The environment perception mechanism based on Transformer is used to fuse the spatiotemporal features of the scene perception feature extraction module, the distance features of the human perception feature extraction module, and personal features to obtain high-order social interaction intensity features.

[0107] Interaction discovery module: Reduces the dimensionality of the social interaction intensity features to obtain the interaction intensity matrix between people, thereby discovering social interaction groups in the video.

[0108] The technologies implemented by each module of the system for detecting social interaction groups in videos based on environmental perception networks correspond to the implementation technologies of the steps in the method embodiment for detecting social interaction groups in videos based on environmental perception networks, and will not be repeated here.

[0109] Based on the above technical description, in one specific embodiment, the experiment used the Social CollectiveActivity Dataset (social-CAD), which specifically annotates groups in the Collective Activity Dataset and is consistent with the definition of social interaction groups. Additionally, NBA videos obtained from the internet were used to create a new experimental dataset named Social Basketball Activity Dataset (social-BAD) for detecting social interaction groups in the videos. Furthermore, these two datasets were combined to form the Video SocialInteraction Dataset (VSID), which was also used to verify the method of this application.

[0110] Experiments have shown that the method in this example can predict the interaction groups in the video by using the obtained interaction intensity matrix and then predicting the interaction matrix after threshold binarization. Table 1 shows the experimental results.

[0111] Table 1, where P is precision, R is recall, F1 is F1-score, Acc is accuracy, w / o roi refers to removing the region of interest, and w / ow g "w / o scene" means to remove the bounding box, while "w / o scene" means to remove the scene environment.

[0112]

[0113] In summary, this embodiment provides a psychologically guided environmental perception network for discovering social interaction groups, guided by the influence of the environment on social interaction in psychology. This network uses two types of environmental visual cues to represent factors influencing social interaction and extracts corresponding visual features. A Transformer-based environmental perception mechanism is used to fuse visual features from both types of environments and construct interaction intensity features. An interaction discovery module processes these intensity features to predict the intensity of interactions between individuals. The entire environmental perception network is optimized using an interaction constraint loss function, which is a critical interaction loss function and a smoothing Fo. β A linear combination of loss functions. Experiments conducted on the Social Collective Activity Dataset, Social Basketball Activity Dataset, and Video Social Interaction Dataset demonstrate the reliability of the proposed method, showing that modeling social interactions in computer vision, inspired by psychology, is an effective approach.

[0114] It should be noted that the steps in the method provided in this application can be implemented using the corresponding modules, devices, units, etc. in the system. Those skilled in the art can implement the steps of the method by referring to the technical solution of the system. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0115] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided in this application in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided in this application can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0116] The specific embodiments of this application have been described above. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for discovering social interaction groups in videos based on environmental perception networks, characterized in that, include: An environment perception network is provided, which designs two types of environmental visual cues to represent factors influencing social interaction: scene-aware visual cues and human-aware visual cues. Each type of visual cue consists of a visual representation and a feature extraction network, wherein: The scene-aware visual cues are obtained by using a 3D network to extract features from the input video segment as a visual representation, and by using an encoder to extract spatiotemporal features. The aforementioned human-perceived visual cues utilize bounding boxes and regions of interest as visual representations of posture openness, orientation, and position that influence interaction in psychology; a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the regions of interest. The extracted distance features and personal features serve as a visual representation of the human environment. A Transformer-based environment perception mechanism is used as a feature fusion network to fuse spatiotemporal features, distance features, and personal features to obtain high-order social interaction intensity features. By reducing the dimensionality of the social interaction intensity features, the interaction intensity matrix between people is obtained, thereby discovering social interaction groups in the video; The aforementioned method employs a Transformer-based environment perception mechanism as a feature fusion network to fuse spatiotemporal features, distance features, and personal features to obtain high-order social interaction strength features, including: Calculate human perception and interaction relationships Using the distance feature As the initial ; Using graph-guided updates, for the personal characteristics renew; Using a Transformer encoder, combined with the spatiotemporal features ST, the personal features are... renew; Perform a negative exponential operation on the updated personal characteristics to obtain paired relationships. ; The final human-perceived interaction relationship is obtained using a moving average method. ; The interaction intensity feature I is obtained by calculating the outer product of the human-perceived interaction relationship and the spatiotemporal features. The step of reducing the dimensionality of the social interaction intensity features to obtain the interaction intensity matrix between people includes: using affine transformation and symmetry operations to reduce the dimensionality of the social interaction intensity features and convert them into an interaction intensity matrix R; The step of reducing the dimensionality of the social interaction strength features using affine transformations and symmetry operations to transform them into an interaction strength matrix R includes: We reduce the dimensionality of high-dimensional social interaction strength features and use a softmax operation to constrain them to the range of 0-1, i.e.: ,in and All are learnable parameters, and X is the probability pattern matrix; A symmetry operation is used to ensure the symmetry of the interaction strength matrix, and its diagonal elements are assigned the value 0, thus obtaining the interaction strength matrix R, i.e.: ,in This represents the Hadamard product, where U is a diagonal element that is 0 and the rest are 1. matrix.

2. The method for discovering social interaction groups in videos based on environmental perception networks according to claim 1, characterized in that, The scene-aware visual cues correspond to the scene environment, which is considered from a global perspective, including static targets and dynamic information around a person. The human visual cues refer to the human environment, which is considered from a local perspective, including the appearance and location information of everyone around each person except themselves.

3. The method for discovering social interaction groups in videos based on environmental perception networks according to claim 1, characterized in that, The process of extracting features from the input video segment using a 3D network as a visual representation and extracting spatiotemporal features using an encoder includes: Extracting features from the input video using an I3D network pre-trained on Kinetics-400 As a visual representation of a scene; An encoder consisting of convolutional layers, pooling layers, and temporal averaging operations is used. The spatiotemporal features ST are extracted and used as scene features.

4. The method for discovering social interaction groups in videos based on environmental perception networks according to claim 1, characterized in that, The method utilizes bounding boxes and regions of interest as visual representations of pose openness, orientation, and position that influence interaction in psychology, wherein: By using each person's bounding box as a visual representation of distance information that influences interaction, a distance feature extractor consisting of relative position encoding and a neural network is used to obtain the distance features between any two people. , By utilizing each person's region of interest as a visual representation of the openness and orientation of posture that influence interaction, and extracting features from this region using a personal feature extractor composed of a spatiotemporal encoder, the individual characteristics of each person can be obtained. .

5. The method for discovering social interaction groups in videos based on environmental perception networks according to claim 1, characterized in that, Also includes: After obtaining the interaction intensity matrix, a pairwise interaction constraint loss function is established for the interaction intensity matrix between each pair of people in the video, and the interaction critical loss is used. To increase differentiation, use smoothing loss function This is to alleviate the inaccurate classification caused by category imbalance.

6. The method for discovering social interaction groups in videos based on environmental perception networks according to claim 5, characterized in that, The interaction strength matrix between each pair of people in the video is used to establish a pairwise interaction constraint loss function, which utilizes the interaction critical loss. To increase differentiation, use smoothing loss function To mitigate classification inaccuracies caused by class imbalance, the following measures are taken: The hyperbolic tangent function is used to perform thresholding on the interaction intensity matrix R, i.e. ,in Let be the threshold, c be a hyperparameter, and then use the given labels in the dataset (i.e., people belonging to the same group are assigned the same number) to obtain the true interaction matrix G, and calculate the loss function as follows: ,in This indicates whether there is interaction between any two people. People belonging to the same group have interaction, which is set to 1; otherwise, it is set to 0. This represents the interaction strength matrix between any two people after processing with the hyperbolic tangent function, ranging from 0 to 1; smooth loss function for: ； in ; The final pairwise interaction constraint loss function is obtained by linearly combining the two loss functions: ,in This is a hyperparameter.

7. A system for detecting social interaction groups in videos based on an environmentally aware network, employing the method for detecting social interaction groups in videos based on an environmentally aware network as described in any one of claims 1-6, characterized in that, include: An environmental perception network is designed with two types of environmental perception visual cues to represent factors influencing social interaction, wherein the two types of environmental perception visual cues include scene perception visual cues and human perception visual cues. Scene-aware visual cue module: It uses a 3D network to extract features from the input video segment as a visual representation, and uses an encoder to extract spatiotemporal features; The Human Perception Visual Cue Module utilizes bounding boxes and regions of interest (ROIs) as visual representations of pose openness, orientation, and position that influence interaction in psychology. Specifically, a distance feature extractor is used to extract distance features from the bounding boxes, and a personal feature extractor is used to extract personal features from the ROIs. The extracted distance and personal features serve as a visual representation of the human environment. Environmental perception mechanism module: The environment perception mechanism based on Transformer is used as a feature fusion network to fuse the spatiotemporal features extracted by the scene perception visual cues, the distance features extracted by the human perception visual cues, and personal features to obtain high-order social interaction intensity features. Interaction discovery module: Reduces the dimensionality of the social interaction intensity features to obtain the interaction intensity matrix between people, thereby discovering social interaction groups in the video.

Images