Method and device for identifying multivariate social relationships in video, electronic equipment and storage medium

Abstract

The invention discloses a method and a device for identifying multivariate social relationships in a video, electronic equipment and a storage medium, features extracted from video frames are fused by using an uncertain sensing score, the fused features and mapped role body features are used as node features, edges are established, an adjacent matrix is generated, and the adjacent matrix is used for identifying the multivariate social relationships in the video. Forming fragment sub-graphs together with node features; the fragment sub-graphs are sent into a dynamic graph convolutional network, and role representation is enhanced; updating the enhanced global features and sending the updated global features to block local downsampling; performing mean pooling on the down-sampled global features, and splicing the enhanced and updated node features of each relation pair in the same segment and the averaged global features to generate a segment-level example; performing mean pooling on all the fragment-level examples of the same relation pair to obtain video-level examples; and calculating the similarity between the text features of the tags and the examples, wherein the result is the category of the multi-element social relationship. According to the invention, the accuracy of identifying the social relationship in the video is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video semantic understanding, and specifically relates to a method for identifying multiple social relationships in a video. Background Art

[0002] Identifying social relationships in videos often uses multimodal information such as visual, audio, and text in the video, as well as various deep learning methods, to infer relationships between characters in the video, such as friends, parent-child relationships, etc. Mining valuable information from massive video data helps people quickly understand social relationships in video content. It has the following applications in real-world scenarios: 1) Social behavior analysis: analyzing the structure of teams and organizations, relationships between superiors and subordinates, and intimate relationships between families and groups; 2) Monitoring public safety: by analyzing changes in social relationships in videos, assisting research in areas such as abnormal behavior detection and abnormal event detection.

[0003] Compared with the methods targeting static images, the use of multimodal information and temporal information in videos to identify social relationships in videos has made great progress. At present, the mainstream methods focus on mining fine-grained features in videos, such as bodies, faces, and semantic objects in video frames, and use scene graphs to describe the social status between characters, which improves the performance of the model to a certain extent.

[0004] However, existing technologies do not take emotional information into consideration, do not fully consider spatial semantic information when extracting video action features, and do not treat information at each moment equally, which affects the accuracy of the acquired social relationship-related information. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method for identifying multiple social relationships in a video, thereby improving the accuracy of identifying multiple social relationships in a video.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] A method for identifying multiple social relationships in videos:

[0008] Detect the body and face of the character in the video to be analyzed and identify them to generate an adjacency matrix; extract the character's body features F in the video frame b 、Facial features F f 、Background features F g , text features F t 、Facial emotion features F fe , Background emotional features F ge and text sentiment feature F te , and divide the features into K groups, and then generate a fragment subgraph from the grouped features, the fragment subgraph includes the feature F′ b, F′ ffe , F′ gge and F′ tte As node features and the adjacency matrix A generated based on node co-occurrence; where F′ b It is by F b The node features after mapping, and F′ ffe , F′ gge and F′ tte It is obtained by the following method: projecting the facial features, background features, text features, facial emotion features, background emotion features, and text emotion features corresponding to the characters in each group into the Gaussian distribution of the latent space to generate an uncertainty perception score; using the uncertainty perception score as a weight, respectively fusing the facial features and facial emotion features, background features and background emotion features, text features and text emotion features corresponding to the characters to obtain F′ ffe , F′ gge and F′ tte ;

[0009] The fragment subgraph is fed into a dynamic graph convolutional network to propagate node information, capture the interaction between characters and between characters and scenes, and enhance the representation of characters;

[0010] The enhanced global action features and global audio features are updated and sent to the block local downsampling, and then the downsampled global action features and global audio features are averaged and pooled. Finally, the node features after enhancement and update and the averaged global features of each relationship pair in the same clip are spliced ​​to generate a clip-level example; all the clip-level examples of the same relationship pair are averaged and pooled to obtain a video-level example; the global action features are obtained by using the slowfast method enhanced by spatial information;

[0011] Train a multi-instance classifier, calculate the similarity between the text features of the label and the examples, and select the maximum similarity as the result, that is, identify multiple social relationships.

[0012] Further technical solution, the F b 、F f 、F g 、F t 、F fe 、F ge and F te Perform maximum pooling and map to the same dimension and feature space:

[0013]

[0014] The uncertainty perception score is:

[0015]

[0016] Among them, F′f,t,g,fe,te,ge represents the node features after mapping, D is the corresponding dimension, σ d , σ d denote the d-th component of σ and σ respectively, σ is the variance, and σ represents random noise;

[0017] Taking the uncertainty perception score as the weight, the facial features and facial emotion features corresponding to the character are integrated as follows: The background features and background emotion features corresponding to the fusion role are: The text features and text sentiment features corresponding to the fusion role are: Among them, α f , α fe , α g , α ge , α t Indicates that through S(F′ f,t,g,fe,te,ge ) to obtain the corresponding uncertainty perception score, represents the corresponding features obtained after Gaussian sampling, and μ represents the mean.

[0018] In a further technical solution, the global action feature is obtained by using a slowfast method enhanced by spatial information, specifically: extracting the video feature F to be analyzed c and spatial feature F s , the spatial feature F s The Transformer network based on spatial level attention obtains significant spatial semantic features The video feature F c With the characteristics Splicing, get feature F cs ; The feature F cs It is sent to the multi-scale Transformer network and the feature F is given by the attention mechanism. cs Assign different weights to obtain multi-scale features Z p , the multi-scale feature Z p After pooling, the global action features are obtained.

[0019] A further technical solution is to divide the global action features and global audio features into feature blocks without overlap at a fixed time step, and downsample each feature block individually through a shared convolutional layer.

[0020] A further technical solution is that the single-layer dynamic graph convolutional network is:

[0021] Calculate the score between the source node and its neighboring nodes, assigning different weights to the neighboring nodes:

[0022]

[0023] in: represents the adjacency matrix after processing at layer l, W and b represent learnable parameters, [||] represents the concatenation operation, and represents the characteristics of nodes N1 and N2 in the first layer, Indicates that weights are assigned to different edges;

[0024] The processing includes deleting edges:

[0025]

[0026] Where: τ A Indicates the threshold value;

[0027] In a further technical solution, the characteristics of the node are updated according to the following formula:

[0028]

[0029] Among them, W l is the linear transformation matrix of the lth layer, δ is the activation function, and represents the node features of the lth layer and the l+1th layer, Representation Matrix The degree matrix of , and:

[0030]

[0031] Among them, N1 and N2 represent nodes in the graph.

[0032] A further technical solution is to enhance the representation of characters through dynamic graph convolutional networks:

[0033] F″ b ,F″ ffe ,F″ gge ,F″ tte =DGCN1(A,F′ b ,F′ ffe ,F′ gge ,F′ tte )

[0034] Among them, DGCN1 represents a dynamic graph convolutional network used to transmit node information in the scene graph.

[0035] A further technical solution is to train a multi-instance classifier using the following loss function:

[0036]

[0037] Where s represents the scaling factor, m represents the boundary, θ represents the angle between the two vectors, o represents the category of the relationship pair p, C represents the number of categories, p represents the pth relationship pair consisting of role i and role j, and P represents the number of relationship pairs. It means mapping and normalizing the feature vectors of the clip-level examples and video-level examples of the relationship pair. The text feature vector representing the label is mapped and normalized.

[0038] A device for identifying multiple social relationships in a video, comprising:

[0039] The feature processing module fuses the features extracted from the video frames using the uncertain perception scores;

[0040] In the fragment subgraph generation module, the fused features and the mapped character body features are used as node features, and edges are established to generate an adjacency matrix;

[0041] Dynamic graph convolutional network, which takes a subgraph of a clip as input and enhances the representation of the character;

[0042] The downsampling module updates the enhanced global motion features and global audio features and sends them to the block-based local downsampling;

[0043] The example generation module performs mean pooling on the downsampled global action features and global audio features, and then concatenates the enhanced and updated node features and the averaged global features of each relationship pair in the same clip to generate a clip-level example; all clip-level examples of the same relationship pair are mean pooled to obtain a video-level example;

[0044] The result generation module calculates the similarity between the text features of the tags and the examples, and the result is the category of multiple social relationships.

[0045] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method for identifying multiple social relationships in a video when executing the computer program.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for identifying multiple social relationships in a video.

[0047] The beneficial effects of the present invention are:

[0048] (1) The present invention considers emotional information and spatial semantic information when classifying multiple examples, and can obtain effective features to improve the accuracy of identifying multiple social relationships in videos.

[0049] (2) The present invention projects the facial features, background features, text features, facial emotion features, background emotion features and text emotion features corresponding to the characters in each group of video clips into the Gaussian distribution of the latent space to generate an uncertain perception score; uses the uncertain perception score as a weight to fuse the facial features and facial emotion features, background features and background emotion features, and text features and text emotion features corresponding to the characters; measures the importance of different node features in the clip subgraph and enhances the character-related features.

[0050] (3) The present invention uses a slowfast method with spatial information enhancement to obtain global action features, and fuses the video features with the spatial features sent to the Transformer network based on spatial hierarchical attention. This method can capture significant spatial semantic features and enhance behavioral role discrimination. The global action features and global audio features updated by the Bi-GRU network are divided into feature blocks without overlap according to a fixed time step. The local downsampling layer is divided into blocks through a shared convolutional layer, and downsampling is performed independently in each feature block, making the features more distinguishable in the time dimension. At the same time, this operation can be performed in parallel, greatly reducing the calculation time.

[0051] (4) The present invention uses a dynamic graph convolutional network to capture the interactions between characters and between characters and scenes, and generates multiple examples for each relationship pair; the loss function uses the text features of the labels to obtain more effective social relationship features and more accurately identify multiple social relationships in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the method for identifying multiple social relationships in a video according to the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0054] like Figure 1 As shown, the present invention provides a method for identifying multiple social relationships in a video:

[0055] Detect the character's body and face in all video frames from the input video, and add an ID to the character based on the annotation; evenly divide the input complete video into several video clips of equal length; then group the extracted body features, facial features, background features, text features, facial emotion features, background emotion features and text emotion features according to the time range of the video clip.

[0056] The character's body features, facial features, background features, text features, facial emotion features, background emotion features and text emotion features in each video clip range are pooled to the maximum and mapped to the same dimension and feature space using a fully connected layer. The facial features, background features, text features, facial emotion features, background emotion features and text emotion features corresponding to the characters in each group are projected into the Gaussian distribution of the latent space using an uncertainty perception module to generate an uncertainty perception score, which can measure the importance of different features in the subgraph. Then, the facial features, background features, text features and corresponding emotion features corresponding to the characters are fused as nodes by using the uncertainty perception score as a weight to construct a fragment subgraph adjacency matrix to generate a fragment subgraph. Then, a dynamic graph convolutional network is used to propagate information on the nodes, enhance the representation of the characters, and capture the interactions between characters and between characters and scenes. Then, the node features and global action and audio features are sent to a bidirectional gated recurrent unit to capture temporal information, and then the global action features and global audio features are downsampled, and the downsampled global features are averaged, and the action and audio features are averaged and pooled.

[0057] The body, face, background, text, global action, and global audio features of the two characters in a relationship pair in the same video clip are concatenated to generate a clip-level example, and then all the clip-level examples of a relationship pair are mean-pooled to generate a video-level example; multiple text features of the label are extracted, and the similarity score between the example features and the label text features is calculated as the prediction score, that is, the recognition result.

[0058] Among them, the global action features are extracted by the slowfast method with spatial information enhancement: the significant spatial semantic features are obtained from the spatial features of the video to be analyzed Use slowfast to extract the video feature F of the video to be analyzed c And with the feature Splicing, get feature F cs ; The feature F cs It is sent to the multi-scale Transformer network and the feature F is given by the attention mechanism. cs Assign different weights to obtain multi-scale features Z p ; Multi-scale feature Z p After pooling, the global action features are obtained.

[0059] The specific steps include:

[0060] Step (1), detect the body and face of the character in the video to be analyzed and identify them.

[0061] Step (1.1), use FFmpeg (Fast Forward Mpeg) to read the video frame;

[0062] Step (1.2), use the pre-trained Faster R-CNN (Faster Region-based Convolutional Neural Network) model to detect the character's body in the video frame, and use the pre-trained RetinaFace model to detect the character's face;

[0063] Step (1.3), according to the marked facial range and character ID, add ID to the detected character body and face.

[0064] Step (2), extract the character's body features F in the video frame b 、Facial features F f 、Background features F g , text features F t 、Facial emotion features F fe , Background emotional features F ge , text sentiment feature F te .

[0065] Use the CPN (Cascaded Pyramid Network) model pre-trained on the MS COCO (Microsoft Common Objects in Context) dataset to extract the character’s body features F b ;

[0066] Use the ResNet50 (Residual Network 50) model pre-trained on the MS1MV0 dataset to extract the character’s facial features F f The facial emotion feature F is obtained by fine-tuning the Real-world Affective Faces DataBase (RAF-DB) and Group Affective Database 2.0 (GAF2) emotion databases. fe ;

[0067] Use the ResNet-50 (Residual Network 50) model pre-trained on Place365 data to extract the background features F of the video frame g , and fine-tuned using the Group Affective Database 2.0 (GAF2) sentiment database to obtain the background sentiment feature F ge ;

[0068] Use the pre-trained fastText model to extract text features F t, and fine-tuned using the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) sentiment database to obtain the text sentiment feature F te .

[0069] Step (3), divide the video to be analyzed evenly into K segments, and extract the character body features F in step (2) b 、Facial features F f 、Background features F g , text features F t 、Facial emotion features F fe , Background emotional features F ge and text sentiment feature F te The clips are divided into K groups according to their time range.

[0070] Step (4), generate a fragment subgraph using the features grouped in step (3).

[0071] Step (4.1), the character's body features F in each segment b 、Facial features F f 、Background features F g , text features F t 、Facial emotion features F fe , Background emotional features F ge and text sentiment feature F te Perform maximum pooling to reduce the amount of computation; and use the fully connected layer to transform the character's body features F b 、Facial features F f 、Background features F g , text features F t 、Facial emotion features F fe , Background emotional features F ge and text sentiment feature F te Mapped to the same dimension and feature space:

[0072] F′ b ,F′ f ,F′ g ,F′ t ,F′ fe ,F′ ge ,F′ te =FC(Maxpool(F b ,F f ,F g ,F t ,F fe ,F ge ,F te ))(1)

[0073] Among them, Maxpool represents maximum pooling and FC represents fully connected layer.

[0074] Step (4.2), using the uncertainty perception module, the facial features, background features, text features, facial emotion features, background emotion features, and text emotion features corresponding to the characters in each group are projected into the Gaussian distribution of the latent space to generate an uncertainty perception score:

[0075]

[0076] Among them, F′ f,t,g,fe,te,ge represents the node features after mapping, D is the corresponding dimension, σ d ,∈ d They represent the dth component of σ and ∈ respectively, σ is the variance, and ∈ represents random noise.

[0077] Step (4.3) uses the uncertainty perception score as a weight to fuse the facial features and facial emotion features corresponding to the character: The background features and background emotion features corresponding to the fusion role are: The text features and text sentiment features corresponding to the fusion role are: Among them, α f , α fe , α g , α ge , α t Indicates that through S(F′ f,t,g,fe,te,ge ) to obtain the corresponding uncertainty perception scores, measuring the importance of different features; represents the corresponding features obtained after Gaussian sampling, and μ represents the mean.

[0078] Step (4.4), consider the fused features in step (4.3) as node features, establish edges between different characters, between the body and face of the same character, between the character and the background, and between the background and the text, and generate the adjacency matrix A of the fragment subgraph. The adjacency matrix A is defined as follows:

[0079]

[0080] Where: N1, N2∈N represent nodes in the graph, k∈{1,2,...,K} represents the kth segment, i and j represent different roles respectively. It means that there is an edge between nodes N1 and N2, otherwise there is no edge between the two nodes.

[0081] The node features and the adjacency matrix A are combined together to form a fragment subgraph.

[0082] Step (5), the fragment subgraph obtained in step (4) is fed into the dynamic graph convolutional network to propagate node information, capture the interaction between characters and between characters and scenes, and enhance the representation of characters.

[0083] In step (5.1), a single-layer dynamic graph convolutional network can be expressed as:

[0084] First, considering that different nodes have different interactions, we assign different weights to neighbor nodes by calculating the score between the source node and its neighboring nodes to decide which neighbor node is more important:

[0085]

[0086]

[0087] in: represents the adjacency matrix after processing (weighting, deletion) of the lth layer, express The initial value of , I is the identity matrix, W and b represent learnable parameters, [||] represents the concatenation operation, l represents the lth layer, and Represents the features of different nodes in the lth layer. The node features of the first layer are obtained by step (4.1); Indicates assigning weights to different edges.

[0088] Since multiple characters in the scene may not have direct interactions, establishing edges between the body nodes of characters that do not interact will affect the recognition of social relationships. Therefore, if the edge score between the character bodies is small, delete the edge:

[0089]

[0090] where τ A Indicates the threshold value;

[0091] Then find the matrix The degree matrix The calculation is as follows:

[0092]

[0093]

[0094] The node features are updated according to the following formula:

[0095]

[0096] Among them, W l is the linear transformation matrix of the lth layer, δ is the activation function, and Represents the node features of the lth layer and the l+1th layer.

[0097] Step (5.2), enhance the role representation through dynamic graph convolutional network:

[0098] F″ b ,F″ ffe ,F″ gge ,F″ tte =DGCN1(A,F′ b ,F′ ffe ,F′ gge ,F′ tte ) (9)

[0099] Among them, DGCN1 represents a dynamic graph convolutional network used to transmit node information in the scene graph.

[0100] Step (6), capture the timing information of role interactions.

[0101] Step (6.1), use the slowfast method with spatial information enhancement to obtain the global action feature F in the video to be analyzed v , use the STFT (Short-Time Fourier Transform) model to extract the global audio feature F in the video to be analyzed a ;

[0102] Extract the global action features F in the video to be analyzed v The specific process is:

[0103] Step (6.1.1), use the slowfast method to obtain the video feature F to be analyzed c ;

[0104] Step (6.1.2), feed the video frame in step (1.1) into the pre-trained deep convolutional neural network (Inception-V1) to extract the spatial features F of the video s ;

[0105] Step (6.1.3) transforms the spatial feature F s Obtain salient spatial semantic features through the Transformer network based on spatial level attention

[0106] Step (6.1.4), the video feature F c and salient spatial semantic features Perform feature splicing to obtain feature F cs ,The concatenated features contain richer spatial semantic information, which helps to enhance the ,characterization capability of action features;

[0107] Step (6.1.5), feature Fcs Send it to the encoder of the multi-scale Transformer network to get L t Layer-wise multi-scale feature representation

[0108] Step (6.1.6) converts the pyramid feature Z p After pooling, we get the global action feature F v ;

[0109] Step (6.2), use the bidirectional long short-term memory unit Bi-LSTM (Bidirectional Long Short Term Memory Network) network to update the node features and global action and audio features in step (5.2):

[0110]

[0111] Step (6.3), the feature Input edge feature downsampling to get features

[0112] Common global downsampling, due to the existence of the inductive bias characteristic, will smooth the features, thereby reducing the global feature resolution. In contrast, this embodiment only downsamples independently within each feature block, which can maintain the distinguishability of the features in time series and is more conducive to determining the time boundary of the action. In addition, this operation can be performed in parallel to reduce the calculation time. This embodiment downsamples the features The feature blocks are divided into non-overlapping feature blocks according to a fixed time step, and each feature block is downsampled separately through a shared convolutional layer to maintain the distinguishability of the features in the time dimension. The downsampled features are

[0113]

[0114] Among them, patch-downsample() represents the block local downsampling layer.

[0115] Step (7), using the node features and global actions and audio features in step (6) to generate segment-level examples and video-level examples.

[0116] Step (7.1) performs mean pooling on the global action and audio features after time series processing:

[0117]

[0118] Step (7.2) concatenates the time-sequentially processed body features, facial features, background features, text features, and averaged global action and audio features of each relationship pair in the same segment to generate a segment-level example:

[0119]

[0120] Among them: i and j represent different roles respectively.

[0121] In step (7.3), all segment-level examples of the same relation pair are mean-pooled to obtain video-level examples:

[0122]

[0123] Step (8) trains a multi-instance classifier to identify the category of social relations by obtaining the text features of the labels and calculating the similarity between the text features of the labels and the fragment-level examples and the video-level examples.

[0124] In step (8.1), considering that a single label text cannot fully describe the social relationship of each category, the pre-trained fastText model is used to extract the label and (Q-1) text features with similar semantics to the label as the central vector to obtain the text feature Y of the label.

[0125] In step (8.2), the multi-instance classifier uses a fully connected layer to map the text feature vector of the label and the segment-level example and video-level example feature vector of the relation pair to the same dimension and perform L2 normalization:

[0126]

[0127]

[0128] Where: p∈{1,2,...,P} represents the pth relationship pair (i.e., role i and role j), and P represents the number of relationship pairs.

[0129] Step (8.3) calculates the dot product between the text features of the mapped label and the relationship pair example features as the similarity, obtains the multi-example prediction score, and selects the maximum value as the result, that is, the category of the social relationship:

[0130]

[0131] Among them, ⊙ represents the dot product operation.

[0132] The following loss function is used to train a multi-instance classifier:

[0133]

[0134] Among them, s represents the scaling factor, m represents the boundary, θ represents the angle between the two vectors, o represents the category of the relationship pair p, and C represents the number of categories.

[0135] The present invention also provides a device for identifying multiple social relationships in a video, comprising:

[0136] The feature processing module fuses the features extracted from the video frames using the uncertain perception scores;

[0137] In the fragment subgraph generation module, the fused features and the mapped character body features are used as node features, and edges are established to generate an adjacency matrix;

[0138] Dynamic graph convolutional network, which takes a subgraph of a clip as input and enhances the representation of the character;

[0139] The downsampling module updates the enhanced global motion features and global audio features and sends them to the block-based local downsampling;

[0140] The example generation module performs mean pooling on the downsampled global action features and global audio features, and then concatenates the enhanced and updated node features and the averaged global features of each relationship pair in the same clip to generate a clip-level example; all clip-level examples of the same relationship pair are mean pooled to obtain a video-level example;

[0141] The result generation module calculates the similarity between the text features of the tags and the examples, and the result is the category of multiple social relationships.

[0142] The electronic device provided by the embodiment of the present invention can implement each process of the method for identifying multiple social relationships in a video in the above method embodiment, and can achieve the same beneficial effects.

[0143] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory may also be an external storage device of an electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Of course, the memory may also include both the internal storage unit of the electronic device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the electronic device, such as a program code of a method for identifying multiple social relationships in a video, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0144] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the electronic device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running a program code of a method for identifying multiple social relationships in a video.

[0145] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the method for identifying multiple social relationships in a video provided by an embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0147] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A method for identifying multiple social relationships in videos, Features: Detect the body and face of the character in the video to be analyzed and identify them to generate an adjacency matrix; extract the character's body features F in the video frame b 、Facial features F f 、Background features F g , text features F t 、Facial emotion features F fe , Background emotional features F ge and text sentiment feature F te , and divide the features into K groups, and then generate a fragment subgraph from the grouped features, the fragment subgraph includes the feature F′ b , F′ ffe , F′ gge and F′ tte As node features and the adjacency matrix A generated based on node co-occurrence; where F′ b It is by F b The node features after mapping, and F′ ffe , F′ gge and F′ tte It is obtained by the following method: projecting the facial features, background features, text features, facial emotion features, background emotion features, and text emotion features corresponding to the characters in each group into the Gaussian distribution of the latent space to generate an uncertainty perception score; using the uncertainty perception score as a weight, respectively fusing the facial features and facial emotion features, background features and background emotion features, text features and text emotion features corresponding to the characters to obtain F′ ffe , F′ gge and F′ tte ; The fragment subgraph is fed into a dynamic graph convolutional network to propagate node information, capture the interaction between characters and between characters and scenes, and enhance the representation of characters; The enhanced global action features and global audio features are updated and sent to the block local downsampling, and then the downsampled global action features and global audio features are averaged and pooled. Finally, the node features after enhancement and update and the averaged global features of each relationship pair in the same clip are spliced ​​to generate a clip-level example; all the clip-level examples of the same relationship pair are averaged and pooled to obtain a video-level example; the global action features are obtained by using the slowfast method enhanced by spatial information; Train a multi-instance classifier, calculate the similarity between the text features of the label and the examples, and select the maximum similarity as the result, that is, identify multiple social relationships.

2. The method for identifying multiple social relationships in a video according to claim 1, It is characterized in that The F b 、F f 、F g 、F t 、F fe 、F ge and F te Perform maximum pooling and map to the same dimension and feature space: F′ b ,F′ f ,F′ g ,F′ t ,F′ fe ,F′ ge ,F′ te =FC(Maxpool(F b ,F f ,F g ,F t ,F fe ,F ge ,F te )) The uncertainty perception score is: Among them, F′ f,t,g,fe,te,ge represents the node features after mapping, D is the corresponding dimension, σ d , σ d denote the d-th component of σ and σ respectively, σ is the variance, and σ represents random noise; Taking the uncertainty perception score as the weight, the facial features and facial emotion features corresponding to the character are integrated as follows: The background features and background emotion features corresponding to the fusion role are: The text features and text sentiment features corresponding to the fusion role are: Among them, α f , α fe , α g , α ge , α t Indicates that through S(F′ f,t,g,fe,te,ge ) to obtain the corresponding uncertainty perception score, represents the corresponding features obtained after Gaussian sampling, and μ represents the mean.

3. The method for identifying multiple social relationships in a video according to claim 1, It is characterized in that The global action feature is obtained by using the slowfast method enhanced by spatial information, specifically: extracting the video feature F to be analyzed c and spatial feature F s , the spatial feature F s The Transformer network based on spatial level attention obtains significant spatial semantic features The video feature F c With the characteristics Splicing, get feature F cs ; The feature F cs It is sent to the multi-scale Transformer network and the feature F is given by the attention mechanism. cs Assign different weights to obtain multi-scale features Z p , the multi-scale feature Z p After pooling, the global action features are obtained.

4. The method for identifying multiple social relationships in a video according to claim 3, It is characterized in that The global action features and global audio features are divided into feature blocks without overlap at fixed time steps, and each feature block is downsampled separately through a shared convolutional layer.

5. The method for identifying multiple social relationships in a video according to claim 1, It is characterized in that A single-layer dynamic graph convolutional network is: Calculate the score between the source node and its neighboring nodes, assigning different weights to the neighboring nodes: in: represents the adjacency matrix after processing at layer l, W and b represent learnable parameters, [||] represents the concatenation operation, and Represents the node N in the lth layer 1 、N 2 Features, Indicates that weights are assigned to different edges; The processing includes deleting edges: Where: τ A Indicates the threshold value; The node features are updated according to the following formula: Among them, W l is the linear transformation matrix of the lth layer, δ is the activation function, and represents the node features of the lth layer and the l+1th layer, Representation Matrix The degree matrix of , and: Among them, N 1 、N 2 Represents a node in a graph.

6. The method for identifying multiple social relationships in a video according to claim 5, It is characterized in that Enhancing character representation via dynamic graph convolutional networks: F″ b ,F″ ffe ,F″ gge ,F″ tte =DGCN 1 (A,F′ b ,F′ ffe ,F′ gge ,F′ tte ) Among them, DGCN 1 Represents a dynamic graph convolutional network for transferring node information in a scene graph.

7. The method for identifying multiple social relationships in a video according to claim 1, It is characterized in that The following loss function is used to train a multi-instance classifier: Where s represents the scaling factor, m represents the boundary, θ represents the angle between the two vectors, o represents the category of the relationship pair p, C represents the number of categories, p represents the pth relationship pair consisting of role i and role j, and P represents the number of relationship pairs. It means mapping and normalizing the feature vectors of the clip-level examples and video-level examples of the relationship pair. The text feature vector representing the label is mapped and normalized.

8. A device for implementing the method for identifying multiple social relationships in a video as described in any one of claims 1 to 5, It is characterized in that include: The feature processing module fuses the features extracted from the video frames using the uncertain perception scores; In the fragment subgraph generation module, the fused features and the mapped character body features are used as node features, and edges are established to generate an adjacency matrix; Dynamic graph convolutional network, which takes a subgraph of a clip as input and enhances the representation of the character; The downsampling module updates the enhanced global motion features and global audio features and sends them to the block-based local downsampling; The example generation module performs mean pooling on the downsampled global action features and global audio features, and then concatenates the enhanced and updated node features and the averaged global features of each relationship pair in the same clip to generate a clip-level example; all clip-level examples of the same relationship pair are mean pooled to obtain a video-level example; The result generation module calculates the similarity between the text features of the tags and the examples, and the result is the category of multiple social relationships.

9. An electronic device, It is characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for identifying multiple social relationships in a video as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method for identifying multiple social relationships in a video according to any one of claims 1 to 7 is implemented.

Images