A video entity relationship and interaction recognition method based on multimodal features

Abstract

A method for identifying video entity relationships and interactions based on multimodal features predicts and identifies interactions between entities in a video: the video is divided into medium segments according to the scene, and the medium segments are aggregated upward into long segments and divided downward into short segments. For each medium segment, the extracted entity features, entity pair features, video segment features, audio features, and text features are spliced ​​as the fusion features of the segment, and the average is taken as the feature of the long segment to predict the entity relationship in the long segment. The average feature is connected to each medium segment feature to predict the interaction between entities in the corresponding medium segment. Entity relationships and entity interactions are predicted simultaneously, and the corresponding recognition network is jointly trained. The present invention divides long videos into three lengths of video, predicts entity relationships for long segments, predicts entity interactions and scene states for medium segments, and predicts entity interactions for short segments, and constructs entity relationship graphs and interaction graphs.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and relates to entity relationship detection, entity interaction detection and scene state prediction in videos, and specifically, is a video entity relationship and interaction recognition method based on multimodal features. Background Art

[0002] Long videos generally refer to videos that are longer than half an hour, with plot-based content and a complete storyline, and are generally movies. Relationship analysis and interaction analysis between different entities in long videos contribute to the in-depth understanding of long videos, which often requires inferring hidden information based on known information. Relationship analysis specifically refers to the construction of a (social) relationship diagram between two types of entities, scenes and characters, that appear in the entire film. Interaction analysis is dedicated to constructing an action interaction diagram that appears between character entities in different video clips. The present invention also involves predicting scene states. Various questions about in-depth analysis of videos can be answered through entity relationship diagrams, interaction diagrams, and scene state information.

[0003] Similar work on video understanding includes video summarization, behavior recognition, visual relationship detection, social relationship recognition, and video emotion detection. However, these efforts are generally applicable to short videos and lack in-depth analysis of the relationships and interactions between different entities. For long video analysis, the following problems still exist:

[0004] 1) Short videos have relatively limited content, often consisting of only one scene and a few characters. Existing technologies for short video analysis cannot address multiple entities, including relationships between characters and scenes, and predict interactions.

[0005] 2) It is difficult to combine the analysis of short videos, and it is impossible to predict the relationships and interactions between entities that are not in the same frame.

[0006] 3) The status analysis of short videos is relatively simple and cannot identify and solve some complex situations.

[0007] At the same time, existing analysis methods are mostly suitable for situations with sufficient training samples, while deep relationship analysis tasks for long videos often lack training samples for some relationships. Furthermore, few methods can jointly analyze relationships and interactions between entities. Therefore, existing technologies cannot solve the problem of deep relationship and interaction analysis in long videos. Summary of the Invention

[0008] The problem to be solved by the present invention is that the technology needs to have a deep understanding of long videos, build interactive relationships between entities in long videos, and perform in-depth analysis of the videos.

[0009] The technical solution of the present invention is: a video entity relationship and interaction recognition method based on multimodal features, which uses recognition and tracking means to extract visual features from the film, and combines audio and dialogue features to predict and recognize the interactions between entities in the film: the film is divided into medium-length video clips according to the scene, called mid-segments, and the mid-segments are first aggregated upward into long clips according to the scene, visual and sound models, and divided downward into short clips according to the scene shots. At the same time, the characters and scenes in the mid-segments and short clips are recognized and tracked according to the specified character screenshots and scene screenshots, and the corresponding entity features in the mid-segments are extracted. Entity pair features are extracted for two entities that appear simultaneously in the mid-segment, and the audio and text features of the corresponding mid-segment are derived. For each mid-segment, the extracted entity features, entity pair features, video clip features, audio features and text features are spliced ​​as the fusion features of the segment, and then the features of multiple mid-segments that constitute a long segment are averaged as the features of the long segment to predict the entity relationship in the long segment, and the average feature is connected to each mid-segment feature to predict the interaction between entities in the corresponding mid-segment. The entity relationship and entity interaction are predicted simultaneously, and the corresponding recognition network is jointly trained;

[0010] Directly predicting character interactions from short clips to provide the timing of entity interactions in mid-segment;

[0011] The video, audio and text features of long clips are spliced ​​together to form scene features, and then a model is trained to predict the scene information of the video based on known scene labels.

[0012] Furthermore, long clips are identified and tracked based on characters and scenes, and fused features are obtained in the same way as medium clips. The entity relationships in the long clips are predicted and output as the result of the entity relationships.

[0013] The average of mid-segment features is used as the feature prediction for long segments to predict entity relationships. Joint training with entity interaction prediction for mid-segments yields more accurate training results than training only on entity interactions based on mid-segment features, facilitating entity interaction prediction. Regarding entity relationships, identifying and tracking people and scenes in long segments yields better results than predicting them based on average features. Therefore, the present invention prefers to jointly train entity interactions from mid-segments while simultaneously predicting entity relationships for long segments.

[0014] Furthermore, a recognition network is established to identify entity relationship graphs, entity interaction graphs and scene information in videos. The input of the recognition network includes the mid-segment of the film to be identified, as well as specified scene screenshots, scene names, character screenshots and corresponding character names. The output is the relationship graph, interaction graph and corresponding scene information between the corresponding scenes and characters. Multimodal features are used to jointly train the recognition network. First, the mid-segment is semantically aggregated into a long segment according to the scene, vision and sound models. Then, video, sound and subtitle text features are extracted from the mid-segment. The position where the scene or character appears in the video segment is identified according to the given scene screenshots and character screenshots. The corresponding fusion features are extracted. The fusion features of a group of mid-segments that make up the long segment are averaged as the feature of the long segment to predict the entity relationship in the long segment. The average feature is connected to each mid-segment feature to predict the interaction between entities in the corresponding mid-segment. When training the recognition network, the long segment entity relationship and the mid-segment entity interaction are predicted simultaneously and trained jointly. At the same time, the video, sound and text features of the mid-segment are separately spliced ​​as the scene feature of the mid-segment.

[0015] Among them, small-sample learning is combined with zero-sample learning to simultaneously predict entity relationships in long clips and entity interactions and scene information in medium clips, so as to achieve the purpose of mutual influence and joint learning of relationships, interactions and scene states.

[0016] Relationships predict social relationships, while interactions predict the action interactions between characters. Relationships are generally stable, while interactions are variable, that is, a pair of characters may have only one social relationship in a film, but they embody multiple interactions in different video clips. This is why relationships are predicted based on long video clips, while entity interactions and entity relationships are related but different. The present invention obtains an entity relationship graph and an entity interaction graph for the entire film. Interactions are predicted based on medium / short video clips, and relationships are predicted from long clips. The present invention jointly predicts the relationships, interactions, and video status information between entities through the aggregation and segmentation of video segments and the recognition of scenes and characters, and establishes a joint training network based on the fusion of multiple features of visual, sound, and text features, providing a solution for deep relationship analysis on long videos. After aggregating and segmenting the input video, the visual, sound, and text features are integrated to construct a relationship graph and interaction graph between different entities in the video and predict the scene status of the video. These can be used to answer questions about deep understanding of videos.

[0017] The present invention includes video aggregation and segmentation technology, human tracking technology, face recognition technology, feature matching technology, sentence encoding technology, audio feature extraction technology, joint training technology and small sample learning combined with zero sample learning technology. Video aggregation technology is to cluster medium videos into long videos through feature clustering, and video segmentation technology is to divide medium videos into short videos through shot segmentation. Human tracking can use ordinary object tracking technology or pedestrian tracking technology. Since object tracking technology is prone to lose objects between shots, and pedestrian tracking technology is difficult to track close-ups of people, the present invention combines human tracking technology and face tracking technology, and uses feature matching technology to supplement the lost character trajectory. SURF is a feature matching technology that has both quality and speed advantages. It can adapt to the deformation of template images and match video frames. BERT is a method that has good effects on various tasks in natural language processing. BERT can be used to extract effective features for text. MFCC and LMFE are audio features commonly used in speech-to-text technology. The mid-length video features are averaged as long-length video features to represent entity relationships. The long-length video features are then combined with the mid-length video features to create updated mid-length video features to represent entity interactions. Simultaneously, they are trained and predicted in conjunction with scene features, achieving a joint training effect where entity relationships, interactions, and scene states mutually influence each other. Cosine similarity of long-length video features can be used to determine categories and is commonly used in few-shot and zero-shot learning.

[0018] The beneficial effects of the present invention are: providing a solution for in-depth understanding of movies. Since the movies faced by the present invention are all long, they are also called long videos. Long videos are divided into three lengths of videos, and entity relationships are predicted for long clips, entity interactions and scene states are predicted for medium clips, and entity interactions are predicted for short clips, thereby constructing entity relationship graphs and interaction graphs. Long clips are used to answer entity relationship questions, the interactions of medium clips are used to answer most entity interaction questions, and short clips are specifically used to answer interaction timing questions. The present invention solves the problem of joint prediction of entity relationships, interactions and scene states of long videos, and can cope with zero-sample situations, with good breadth and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Define the input and output and video analysis problem types for the present invention.

[0020] Figure 2 This is the multi-feature based joint training feature network architecture of the present invention.

[0021] Figure 3 The results of the present invention are compared when using different feature and branch combinations. DETAILED DESCRIPTION

[0022] The present invention proposes a joint training method for video depth analysis based on multiple features, which combines multiple features to establish a joint training network for identifying entity relationship diagrams and interaction diagrams in videos, and predicting the scene status of videos. It uses recognition and tracking methods to extract visual features from the film, and combines audio and line features to predict and identify the interactions between entities in the film: the film is divided into medium-length video clips according to the scene, called medium clips. First, the medium clips are aggregated upward into long clips according to the scene, vision and sound models, and divided downward into short clips according to the scene shots. At the same time, the characters and scenes in the medium and short clips are analyzed according to the specified character screenshots and scene screenshots. Line recognition and tracking, extracting corresponding entity features in the mid-segment, extracting entity pair features for two entities that appear simultaneously in the mid-segment, and deriving the audio and text features of the corresponding mid-segment. For each mid-segment, the extracted entity features, entity pair features, video segment features, audio features, and text features are spliced ​​as the fusion features of the segment. The features of multiple mid-segments that constitute a long segment are then averaged and used as the features of the long segment to predict the entity relationships in the long segment. The average feature is then connected to each mid-segment feature to predict the interactions between the entities in the corresponding mid-segment. Entity relationships and entity interactions are predicted simultaneously, and the corresponding recognition network is trained jointly.

[0023] Directly predicting character interactions from short clips to provide the timing of entity interactions in mid-segment;

[0024] The video, audio and text features of long clips are spliced ​​together to form scene features, and then a model is trained to predict the scene information of the video based on known scene labels.

[0025] For predicting entity relationships, the average of mid-segment features is used as a feature for predicting entity relationships in long segments. Jointly training the mid-segment features to predict entity interactions yields more accurate training results than training only the mid-segment features to predict entity interactions, facilitating the prediction of entity interactions. Regarding entity relationships, identifying and tracking people and scenes in long segments yields better results than predicting with average features. Therefore, the present invention prefers to jointly train the entity interaction results from mid-segments while simultaneously predicting entity relationships for long segments.

[0026] The implementation of the present invention is described in detail below.

[0027] like Figure 1As shown, the recognition network input of the present invention includes a mid-length clip of the film to be recognized, as well as a designated scene screenshot, scene name, character screenshot, and corresponding character name. The output is a relationship diagram and interaction diagram between the corresponding scene and character, as well as the corresponding scene information. The output is used to analyze the type of questions in long videos, and these questions will be answered based on the output relationship diagram, interaction diagram, and scene status. The input of the multi-feature joint training network is provided by the dataset. The recognition network is based on joint training of multimodal features: first, the mid-segments are semantically aggregated into long segments with more unified semantics according to scene, visual and sound models, and a group of mid-segments are aggregated into multiple independent long segments. Then, video, sound and subtitle text features are extracted from the mid-segments. The location of the scene or character in the video segment is identified based on the given scene screenshots and character screenshots, and the corresponding fusion features are extracted. The fusion features of a group of mid-segments that make up the long segment are averaged as the feature of the long segment, which is used to predict the entity relationship in the long segment. The average feature is connected to each mid-segment feature to predict the interaction between entities in the corresponding mid-segment. When training the recognition network, the long segment entity relationship and the mid-segment entity interaction are predicted simultaneously and trained jointly. At the same time, the video, sound and text features of the mid-segment are separately spliced ​​as the scene feature of the mid-segment.

[0028] By combining small-shot learning with zero-shot learning, we simultaneously predict entity relationships in long clips and entity interactions and scene information in medium clips. This allows for the mutual influence and joint learning of relationships, interactions, and scene states. Finally, we merge the entity pair relationships in the long clips to generate an entity relationship graph for the entire video.

[0029] like Figure 2 As shown, the implementation of the joint training network based on multimodal features of the present invention specifically includes the following steps:

[0030] 1) Based on multimodal features, the input mid-length segments are aggregated into semantically more unified long segments, and at the same time, they are divided into multiple short segments based on the shots, with each short segment corresponding to a scene shot. The activities between a group of people generally take place in a fixed scene, but the same scene may have multiple shots. The short segments correspond to such scene shots, and the time sequence in the scene is obtained based on the shots. In the present invention, sound features, text features, and visual features are all extracted from the video segment. Specifically:

[0031] 1.1) Divide the mid-length clip into multiple shot segments, or short clips. A shot is an uninterrupted, unedited segment of video. The activities of a group of characters generally occur within a scene and are represented as a continuous aggregation of one or more shots.

[0032] 1.2) Extract features for each mid-segment based on the scene model, visual model, and sound model;

[0033] 1.3) Using the three features extracted in step 1.2), semantically aggregate and merge the shot segments into long segments. Using the three features of each mid-segment, predict whether the mid-segment is the last mid-segment of the long segment, thereby determining the boundaries of each long segment, i.e., the last mid-segment of the segment.

[0034] 1.4) Combine the aggregation results based on the three features in step 1.3):

[0035] E=E p ∪E v ∪E a

[0036] Among them, E is the final set of long fragments, E p is the aggregated boundary set of the video according to the scene model, E v is the aggregated boundary set based on the mid-segment of the visual model, E a It is a collection of aggregated boundaries based on the mid-segment of the sound model.

[0037] 2) Use SURF (Speeded Up Robust Features) feature matching to match scene screenshots with frames in each of the short / medium / long video segments segmented in step 1). For each video segment, the scene with the highest total number of matching points is selected as the scene in which it is located. The scene trajectory is the entire screen over the entire video segment. The total number of matching points between the video and each scene is calculated as follows:

[0038]

[0039] Among them, M l represents the total number of matching points of scene l, represents the number of matching points of the screenshot ls of scene l, S l is a set of screenshots of scene l. The scene corresponding to the video segment is the scene with the highest total number of matching points.

[0040] 3) In each of the short / medium / long video segments segmented in step 1), a combination of human tracking and face detection is used to identify and track people by limiting the intersection-over-union ratio of the detected human bounding box and the recognized face bounding box. SURF feature matching is then used to match the person screenshots with the frames as a supplement to face recognition:

[0041] 3.1) Use the center point tracking method to obtain the trajectory of the human body bounding box;

[0042] 3.2) Using the InsightFace face detection and recognition method, a face bounding box trajectory matching a given person's screenshot is obtained. When there is overlap with the person's bounding box and the intersection-and-union ratio is greater than a threshold, the person's bounding box trajectory is the person's bounding box trajectory. When there is no overlap with the person's bounding box or the intersection-and-union ratio is less than a threshold, the person's bounding box trajectory is estimated as an approximate person bounding box of the face bounding box trajectory. The approximate person bounding box estimation is obtained by presetting the face bounding box in the center of the upper half of the person bounding box and calculating the person's position based on the face bounding box estimation. In this embodiment, the face bounding box is expanded by 2 times in width and 7 times in height to obtain the approximate person bounding box trajectory.

[0043] 3.2) Use SURF feature matching to match unrecognized person screenshots to frames. The trajectory of the person's bounding box in the corresponding frame is used as the trajectory of the person's bounding box. This step involves continuing to identify the target person using SURF feature matching if the target person is not recognized by face detection in the current scene. If multiple person bounding boxes appear in the corresponding frame, the intersection over union (IOU) of the SURF matching area and the person bounding box is calculated. If both exceed a threshold of 0.7, the largest value is taken.

[0044] 4) Input each short / medium / long video segment segmented in step 1) into the pre-trained C3D network to calculate the video features.

[0045] 5) Extract the Mel-frequency cepstral coefficients (MFCCs) and log-Mel-frequency energy (LMFE) features of the audio from each short / medium / long video segment segmented in step 1), and calculate the first-order and second-order differences of the MFCC features and the LMFE features to obtain the sound features of the video segment.

[0046] 6) Automatically generate subtitles for the input video and assign them to each short / medium / long video segment segmented in step 1) based on time. Use the BERT network to generate text features for all feature sentences to be calculated:

[0047] 6.1) Automatically generate video subtitles, such as using autoSub software, or the automatic subtitle generation function of Alibaba Cloud or YouTube;

[0048] 6.2) Map each subtitle to a subtitle segment according to the subtitle time period;

[0049] 6.3) When a subtitle sentence is longer than 15 words or the interval between it and the next sentence is longer than 0.5 seconds, it is considered as a feature sentence to be calculated. When a subtitle sentence is less than 15 words and the interval between it and the next sentence is less than 0.5, this sentence is connected to the next sentence to form a new sentence. When the new sentence is longer than 15 words or the interval between it and the next sentence is longer than 0.5 seconds, it is considered as a feature sentence to be calculated. Otherwise, it continues to be connected to the next sentence.

[0050] 6.4) All feature sentences to be calculated use the BERT (Bidirectional Encoder Representations from Transformers) network to generate text features.

[0051] 7) In each short / medium / long video segment segmented in step 1), the visual features of the entities are calculated using the pre-trained C3D network based on the bounding box trajectories of the scene and character entities in steps 3) and 4), and the visual features of the entity pairs are also calculated based on the joint bounding box trajectory of each two entities.

[0052] 8) Connect the video features of step 4), the sound features of step 5), the text features of step 6), the visual features of the two entities in step 7), and the visual features of the entity pair in step 7) corresponding to each entity pair in the middle segment as the interaction features of the entity pair.

[0053] 9) For the same pair of entities in all the medium segments that make up a long segment, the entity pair interaction features obtained in step 8) are averaged to form the relationship feature of the entity pair. The entity pair relationship feature is then concatenated with each interaction feature to form the final interaction feature.

[0054] 10) For each mid-length segment, the video features obtained in step 4), the sound features obtained in step 5), and the text features obtained in step 6) are connected as scene features of the mid-length segment;

[0055] 11) Since the present invention adopts a training method that combines small sample learning with zero sample learning, it introduces the features of classification labels, that is, the features of relationship predicates, interaction predicates and scene state descriptions, rather than directly performing full-connection classification on entity pair features. The BERT model is used to generate semantic features for relationship predicates, interaction predicates and scene states, and the relationship features, interaction features and scene features obtained in steps 9) and 10) are converted into semantic expressions. By calculating the similarity between the semantic expressions of the relationship, interaction and scene features and the semantic features of the target relationship, interaction and scene state labels, it is determined what kind of relationship the entity pair is in, what kind of interaction it is, and what state the video segment is in. The details are as follows:

[0056] 11.1) Convert the relationship features, interaction features, and scene features obtained in steps 9) and 10) into semantic space through linear transformation;

[0057] 11.2) Generate corresponding semantic features for predefined relationship predicates, interaction predicates, and scene status descriptions using the BERT model. Relationship predicates, interaction predicates, and scene status are all predefined, and each relationship predicate / interaction predicate / scene status has a sentence description. Generate corresponding semantic features based on this sentence description.

[0058] 11.3) Based on the semantic features obtained in 11.1) and 11.2), the loss function is calculated when combining small-sample and zero-shot learning training:

[0059]

[0060]

[0061]

[0062]

[0063] Among them, Lr is the relationship loss function, Li is the interaction loss function, Ls is the scene state loss function, and L is the total loss function. β represents the semantic space features of relationship / interaction / scene state features, γ represents the semantic features of real relationship predicates / interaction predicates / scene states, P represents the set of non-real relationships in long segments, Q represents the set of interactions in medium segments, H represents the set of scene states in medium segments, and μ i Represents the features of non-real relationship / interaction / scene state i, p represents the number of non-real relationships, q represents the number of non-real interactions, h represents the number of non-real scene states, N represents the set of mid-segments that make up the long segment, n represents the number of mid-segments that make up the long segment, Lij is the interaction loss of the j-th mid-segment, and Lsj is the scene state loss of the j-th mid-segment;

[0064] 11.4) Calculate the similarity between the semantic expression of the relationship feature / interaction feature / scene state feature and the semantic features of the relationship predicate / interaction predicate / scene state as the probability of the entity pair in each relationship / interaction, and the probability of the video segment in each scene state, thereby determining the relationship / interaction between the entity pair and the scene state of the video segment.

[0065] This paper uses small-sample and zero-shot learning to introduce relational predicates, interaction predicates, and scene state feature training. The BERT model is used to generate semantic features. 11.3) is the training phase, and 11.4) is the prediction phase.

[0066] 12) Based on the entity relationships / interactions / scene states predicted in step 11), select those with a similarity measure greater than 0 to predefined relationship categories / interaction categories / scene state categories as relationship / interaction / scene state candidates, and construct a preliminary entity relationship graph and interaction graph, as well as scene states. With similarity ranging from -1 to 1, select relationships with a similarity greater than 0 to predefined labels as predicted candidates to construct the relationship graph / interaction graph / scene state. When constructing the relationship graph and interaction graph, convert the relationship / interaction triple list into a relationship / interaction graph, i.e., construct a graph structure with entities (scenes, characters) as nodes and relationships / interactions as edges.

[0067] 13) Based on step 12), adjust the relationship prediction probability according to the entity type, name and relationship type, and supplement the relationship between entities:

[0068] 13.1) Adjust the predicted probabilities of different relationship categories based on whether the entity pair is "scene-person", "scene-scene", "person-scene" or "person-person";

[0069] 13.2) Adjust the predicted probabilities of different relationship categories based on whether the entity name contains kinship terms, occupational titles, or common scenario type names.

[0070] 14) Fill the video knowledge graph based on the long-segment entity relationship graph constructed in step 13) to obtain the entity relationship graph for the entire video, which is used to answer the relationship between entities and find the relationship path between two entities; then, based on the entity interaction graph and scene state graph constructed in step 12), obtain the entity interaction situation and scene state in the middle segment, and directly answer the interaction questions between entities and scene recognition questions.

[0071] 14.1) For relational knowledge graph filling and interactive knowledge graph filling for a given mid-video segment, find the predicted entity-relationship graph / interaction graph to fill in the missing knowledge graph nodes;

[0072] 14.2) For the video segment identification problem of a given interaction set, find the video segment containing the target interaction set;

[0073] 14.3) For relational answers, traverse the candidate answers given. If the answer matches the predicted entity relationship diagram, it is the correct answer. If all answers do not match the predicted entity relationship diagram, select the answer with the highest probability.

[0074] 14.4) For the scene selection problem, for each candidate mid-length video segment, its predicted interaction and scene state are matched with the given natural language description by keywords. The one with the highest matching rate is the correct answer.

[0075] 14.5) For the scene state selection problem, for a given mid-segment, the candidate scene state with the highest probability is the correct answer.

[0076] 14.6) For the entity relationship path, depth-first traverse the predicted entity relationship graph to obtain the relationship path between the two entities.

[0077] 15) For each entity pair in the short clip, the video features, sound features, text features, visual features of the entity, and visual features of the entity pair are connected as the interactive features of the entity pair;

[0078] 16) Using the same method as step 11), predict the interaction relationship between the entity pairs in the short segment by combining small-sample learning with zero-shot learning;

[0079] 17) Based on the entity interactions predicted in step 16), the final short segment entity interaction graph is obtained by following step 12). The entity interactions in all short segments that make up the same mid-segment are sorted chronologically to answer the entity interaction order question;

[0080] 17.1) For a given interaction in a given mid-segment, traverse the set of short segments that make up the mid-segment and determine which short segment the interaction is in.

[0081] 17.2) Same as step 17.1), determine the video segment locations of all candidate interactions

[0082] 17.3) The position of the video segment closest to step 17.1) in step 17.2) is the previous / next interaction of the target interaction.

[0083] This paper is implemented on the HLVU dataset. To verify the effectiveness of jointly training relationships, interactions, and scene states, we compared different combinations of the combined branches and their features. The metric used is the recall rate when the number of true samples is taken.

[0084] Figure 3 The three Recalls in the table header represent the recall rate of relationship prediction. R , Recall rate of interactive prediction I And scene state prediction recall rate Recall S .

[0085] Figure 3 The 1st to 5th rows in the table are:

[0086] Relationship branch R using the average of interaction features as the relationship feature s The interaction branch I and the scene state branch S without incorporating entity pair features as scene features are jointly trained.

[0087] Relationship branch R using long segment features as relationship features ss The relationship recall rate of separate training; the interaction recall rate of separate training of interaction branch I; the scene state recall rate of separate training of scene state branch S without incorporating entity pair features as scene features.

[0088] Relationship branch R using long segment features as relationship features ss The interaction branch I and the scene state branch S without incorporating entity pair features as scene features are jointly trained.

[0089] Relationship branch R using the average of interaction features as the relationship feature s Add the interaction branch I and the scene state branch S that incorporates entity pair features as scene features u Joint training.

[0090] Relationship branch R using long segment features as relationship features ss Add the interaction branch I and the scene state branch S that incorporates entity pair features as scene features u Joint training.

[0091] It can be seen that the combined structure of the present invention has advantages in predicting both character relationships and character interactions. Based on the combined architecture, using the average features of medium segments has a better impact on predicting interactions, while using the features of long segments themselves has a better impact on predicting relationships. This is because the features of long segments are based on the entire video, which can introduce unnecessary noise to individual medium segments. For predicting scene states, separate training is more effective.

[0092] Figure 3 In the experiments shown, in the joint architecture, adding person-pair features to the scene state branch revealed a decrease in performance. This suggests that the scene state is based on the entire scene and is less relevant to the interactions between entities. Furthermore, since the dataset contains many clips with only scene states and no interactions, training only on scene states related to entity interactions results in a significant loss of samples, thus impacting prediction recall. Therefore, training the scene branch separately is preferable.

Claims

1. A video entity relationship and interaction recognition method based on multimodal features, characterized by Visual features are extracted from the film using recognition and tracking methods, and combined with audio and dialogue features, the interactions between entities in the film are predicted and identified: the film is divided into medium-length video clips according to the scene, called mid-segments. The mid-segments are first aggregated upward into long clips based on the scene, visual and sound models, and divided downward into short clips based on the scene shots. At the same time, the characters and scenes in the mid-segments and short clips are identified and tracked based on the specified character screenshots and scene screenshots, and the corresponding entity features in the mid-segments are extracted. Entity pair features are extracted for two entities that appear simultaneously in the mid-segment, and the audio and text features of the corresponding mid-segment are derived. For each mid-segment, the extracted entity features, entity pair features, video clip features, audio features and text features are spliced ​​as the fusion features of the clip. The features of multiple mid-segments that constitute a long clip are then averaged as the features of the long clip to predict the entity relationships in the long clip, and the average features are connected to each mid-segment feature to predict the interactions between entities in the corresponding mid-segment. Entity relationships and entity interactions are predicted simultaneously, and the corresponding recognition network is trained jointly. Directly predicting character interactions from short clips to provide the timing of entity interactions in mid-segment; The video, audio, and text features of long clips are spliced ​​together to form scene features. Based on the known scene labels, a model is trained to predict the scene information of the video. A recognition network is established to identify entity relationship graphs, entity interaction graphs, and scene information in videos. The input of the recognition network includes a mid-length clip of the video to be identified, as well as a specified scene screenshot, scene name, character screenshot, and corresponding character name. The output is a relationship graph and interaction graph between the corresponding scene and character, as well as the corresponding scene information. The recognition network is jointly trained using multimodal features, including the following steps: 1) Based on multimodal features, the input medium-length segments are semantically aggregated into long segments, and then divided into multiple short segments according to the shots of the scene, with each short segment corresponding to a shot of a scene; 2) Based on the given scene screenshot, use SURF feature matching to match the scene screenshot to the frame in each short / medium / long segment obtained in step 1). The scene with the highest total number of matching points is selected as the scene corresponding to the video segment. The scene trajectory is the entire segment in time and the image area of ​​each frame in the segment in space. 3) Based on the given person screenshots, use body tracking and face detection methods to identify and track the person in each short / medium / long segment obtained in step 1). Furthermore, use SURF feature matching to match the person screenshots with the frames as a supplement to face recognition to obtain the person's entity bounding box and its trajectory within the scene. 4) Average sampling of each short / medium / long segment obtained in step 1) is performed to obtain a set of key frames, which are then input into the pre-trained C3D network to calculate the video features; 5) Extracting Mel-frequency cepstral coefficients (MFCCs) and log-Mel-frequency energy (LMFE) features from each short / medium / long segment obtained in step 1), and calculating the first-order and second-order differences of the MFCC and LMFE features to obtain sound features; 6) Automatically generate subtitles for the input video and use the BERT network to generate text features for each short / medium / long segment obtained in step 1) according to the timeline; 7) Calculate the visual features of the entities using the pre-trained C3D network based on the scene trajectory and the bounding box trajectory of the person entity in steps 2) and 3) to obtain entity features, and similarly calculate the visual features of the entity pairs based on the joint bounding box trajectory of each two entities to obtain entity pair features; 8) The video features, sound features, text features, visual features of the entities, and visual features of the entity pairs related to each entity pair in the segment are connected as the interaction features of the entity pair; 9) All interaction features obtained in 8) for the same entity pair in all the mid-length segments that make up a long segment are averaged to form the relationship feature of the entity pair. The relationship feature of the entity pair is then concatenated with each interaction feature to form the final interaction feature. 10) Connecting the video features, sound features, and text features of the mid-segment as the scene features of the mid-segment; 11) Convert the relational, interaction, and scene features obtained in 8), 9), and 10) into a semantic space. Combined with the semantic features generated by the BERT model for the relational predicate, interaction predicate, and scene state, simultaneous prediction is performed through a combination of small-shot learning and zero-shot learning. This method jointly learns the relationship between the entity pairs in the long segment, the interaction relationship between the entity pairs in the medium segment, and the scene state in the medium segment. 12) Based on the entity relationships / interactions / scene states predicted in step 11), select those with a similarity measure greater than 0 with predefined relationship categories / interaction categories / scene state categories as relationship / interaction / scene state candidates, and construct a preliminary entity relationship graph, interaction graph, and scene state graph; 13) Based on step 12), adjust the relationship prediction probability according to the type, name, and type of the entity, and supplement the relationship between the entities; 14) Fill the video knowledge graph based on the long-segment entity relationship graph constructed in step 13) to obtain the entity relationship graph for the entire video, which is used to answer the relationship between entities and find the relationship path between two entities; then, based on the entity interaction graph and scene state graph constructed in step 12), the entity interaction status and scene state in the medium segment are obtained to directly answer the interaction questions between entities and scene recognition questions; 15) For each entity pair in the short video segment, the video features, sound features, text features, visual features of the entity, and visual features of the entity pair are connected as the interactive features of the entity pair; 16) Convert the interaction features obtained in 15) into a semantic space, combine them with the semantic features generated by the BERT model for the interaction predicate, and predict the interaction relationship between the entity pairs in the short video segment through a combination of small-shot learning and zero-shot learning; 17) Based on the entity interactions predicted in step 16), the final short video segment entity interaction graph is obtained by following steps 12) and 13), and the entity interactions in all short video segments that make up the same mid-length segment are sorted in chronological order to answer the question of the entity interaction order.

2. The method for identifying video entity relationships and interactions based on multimodal features according to claim 1, characterized in that Long clips are identified and tracked based on characters and scenes, and fused features are obtained in the same way as medium clips. The entity relationships in the long clips are predicted and output as the result of the entity relationships.

3. The method for identifying video entity relationships and interactions based on multimodal features according to claim 1, wherein Calculation method of the long fragment aggregated from the middle fragment in step 1): 1.1) Extract features from the input mid-segment based on the scene model, visual model, and sound model; 1.2) Using the three features extracted in step 1.1), merge the medium segments into long segments; 1.3) Combine the aggregation results based on the three features in step 1.2): E=E p ∪E v ∪E a Among them, E is the set of long segment segmentation boundaries on the final video, E p is a set of long segment boundary segments based on the scene model, E v is a set of long segmentation boundaries according to the visual model, E a It is a set of boundaries for segmenting long segments based on sound models.

4. The method for identifying video entity relationships and interactions based on multimodal features according to claim 1, wherein The total number of matching points between the segment and each scene in step 2) is calculated as follows: Among them, M l represents the total number of matching points of scene l, represents the number of matching points of the screenshot ls of scene l, S l is a set of screenshots of scene l, and the scene corresponding to the video segment is the scene with the highest total number of matching points.

5. The method for identifying video entity relationships and interactions based on multimodal features according to claim 1, wherein Step 11) is specifically as follows: 11.1) Convert the relationship features, interaction features, and scene features obtained in 8), 9), and 10) into semantic space through linear transformation; 11.2) Generate semantic features of predefined relationship predicates, interaction predicates, and scene state descriptions through the BERT model; 11.3) Based on the semantic features obtained in 11.1) and 11.2), the loss function is calculated when combining small-sample and zero-shot learning training: Among them, Lr is the relationship loss function, Li is the interaction loss function, Ls is the scene state loss function, L is the total loss function, β represents the semantic space features of relationship / interaction / scene state features, γ represents the semantic features of real relationship predicate / interaction predicate / scene state, P represents the set of non-real relationships in long clips, Q represents the set of interactions in medium clips, H represents the set of scene states in medium clips, and μ i Represents the features of non-real relationship / interaction / scene state i, p represents the number of non-real relationships, q represents the number of non-real interactions, h represents the number of non-real scene states, N represents the set of mid-segments that make up the long segment, n represents the number of mid-segments that make up the long segment, Lij is the interaction loss of the j-th mid-segment, and Lsj is the scene state loss of the j-th mid-segment; 11.4) Calculate the similarity between the relationship / interaction / scene state features and the relationship predicate / interaction predicate / scene state semantic features as the probability of the entity pair relationship / interaction and scene state in the corresponding video segment, and then determine the relationship / interaction and scene state between the entity pairs.

Images