Deep video understanding methods, apparatus, devices, and readable storage media

Abstract

This application relates to a deep video understanding method, apparatus, device, and readable storage medium, belonging to the field of video understanding technology. The method includes: acquiring text features corresponding to each slice in the video to be understood; extracting features from each slice to obtain the overall features of each slice; performing face detection and tracking on people in each slice to generate a tracking sequence video, and extracting features from the tracking sequence video to obtain person tracking video features; determining the location type corresponding to each slice based on a residual network and the overall slice features; randomly combining people and locations in each slice based on the person tracking video features and the location type to obtain the target features of each slice; concatenating the text features, overall slice features, and target features to obtain the multimodal features of each slice; and predicting the interactions and relationships between people and between people and locations based on the multimodal features. This application can effectively improve the accuracy of prediction results such as relationships and interactions.

Description

Technical Field

[0001] This application relates to the field of video understanding technology, and in particular to a deep video understanding method, apparatus, device, and readable storage medium. Background Technology

[0002] Video understanding is a technology that automatically identifies attributes such as locations, identities of people, interactions, and relationships within a given video. The development of deep video understanding algorithms has benefited many fields, such as generating audio descriptions for movies for the visually impaired, intelligent fast-forwarding of videos, and plot clustering of related characters.

[0003] Because in-depth video understanding of film and television works is a comprehensive problem, it is necessary to predict relationships and interactions centered on characters. However, the actions of characters in film and television works often have a lot of overlap. For example, two people eating, talking, or exercising together could be friends, parents and children, or spouses. However, related technologies often rely solely on visual features to predict relationships and interactions between characters, resulting in low accuracy of the predictions. Summary of the Invention

[0004] This application provides a deep video understanding method, apparatus, device, and readable storage medium to solve the problem of low accuracy in prediction results caused by relying solely on visual features for prediction of relationships, interactions, etc. in related technologies.

[0005] Firstly, a deep video understanding method is provided, including the following steps:

[0006] Obtain the text features corresponding to each slice in the video to be understood;

[0007] Feature extraction is performed on each slice to obtain the overall slice features corresponding to each slice;

[0008] Face detection and tracking are performed on the people in each slice to generate tracking sequence videos. Feature extraction is then performed on the tracking sequence videos to obtain the features of the people tracking videos.

[0009] The location type corresponding to each slice is determined based on the residual network and the overall characteristics of the slice;

[0010] Based on the video features of people tracking and the location type, people and locations in each slice are randomly combined to obtain the target features corresponding to each slice. The target features include people-people pair features and people-location pair features.

[0011] The text features, overall features of each slice, and target features corresponding to each slice are concatenated to obtain the multimodal features corresponding to each slice.

[0012] Based on the multimodal features, predictions are made regarding the interactions and relationships between people and / or between people and locations in the video to be understood, and prediction results are obtained.

[0013] In some embodiments, there is a mapping relationship between the slices and various scenes in the video to be understood. The step of predicting the interactions and relationships between people and / or people and locations in the video to be understood based on the multimodal features, to obtain prediction results, includes:

[0014] Based on the mapping relationship, the multimodal features of all slices in the target scene where the target slice is located are subjected to average pooling to obtain the target scene splicing features;

[0015] Based on the target scene splicing features, the relationships between people and / or between people and locations in the target scene are predicted to obtain relationship prediction results;

[0016] Based on the target multimodal features corresponding to the target slice and the target scene stitching features, the interactions between people and / or between people and locations in the target slice are predicted to obtain the interaction prediction results.

[0017] In some embodiments, predicting the relationships between people and / or between people and locations in the target scene based on the target scene stitching features to obtain relationship prediction results includes:

[0018] Substitute the splicing features of the target scene into the first calculation formula to obtain the score of the relationship between people or between people and locations in the target scene.

[0019] The target relationship corresponding to the highest relationship score is taken as the relationship prediction result for the target scenario.

[0020] The first calculation formula is:

[0021]

[0022] In the formula, s R (V, r) represents the relation score of person-person pair p or person-location pair p for relation r in the target scene, where V represents the slice set corresponding to the target scene, Φ R This represents the splicing features of the target scene. and Both represent network parameters, σ r This represents the activation function.

[0023] In some embodiments, the step of predicting the interactions between people and / or between people and locations in the target slice based on the target multimodal features corresponding to the target slice and the target scene stitching features, to obtain the interaction prediction result, includes:

[0024] Substitute the target multimodal features and target scene stitching features corresponding to the target slice into the second calculation formula to obtain the interaction scores between people or between people and locations in the target slice.

[0025] The interaction relationship corresponding to the highest interaction score is used as the interaction prediction result of the target slice;

[0026] The second calculation formula is:

[0027]

[0028] In the formula, s I (v, V, a) represents the interaction score of person-person pair p or person-location pair p on target slice v for interaction a, Φ I Φ represents the target's multimodal features. R This represents the splicing features of the target scene. W I1 B I1 and Both represent network parameters, σ a This represents the activation function.

[0029] In some embodiments, the step of performing face detection and tracking on the people in each slice to generate a tracking sequence video includes:

[0030] Face detection is performed on the characters in each scene of the video to determine the identity information of each character;

[0031] Track the characters in each scene and generate multiple character tracking sequences;

[0032] Based on the identity information, multiple person tracking sequences are merged to obtain a target tracking sequence;

[0033] Based on the mapping relationship, the target tracking sequence, and the start and end times of the slice, a tracking sequence video corresponding to each slice is generated.

[0034] In some embodiments, after the step of predicting the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, the method further includes:

[0035] A knowledge graph corresponding to the video to be understood is generated based on the prediction results.

[0036] Secondly, a deep video understanding device is provided, including a feature extraction unit, a combination unit, a stitching unit, and a prediction unit;

[0037] The feature extraction unit is used to obtain the text features corresponding to each slice in the video to be understood; to extract features from each slice to obtain the overall features of each slice; to perform face detection and tracking on the people in each slice to generate a tracking sequence video, and to extract features from the tracking sequence video to obtain the people tracking video features; and to determine the location type corresponding to each slice based on the residual network and the overall features of the slice.

[0038] The combination unit is used to randomly combine people and locations in each slice based on the video features of person tracking and the location type to obtain the target features corresponding to each slice. The target features include person-person pair features and person-location pair features.

[0039] The splicing unit is used to splice the text features, overall features of the slice, and target features corresponding to each slice to obtain the multimodal features corresponding to each slice.

[0040] The prediction unit is used to predict the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, and to obtain the prediction results.

[0041] In some embodiments, there is a mapping relationship between the slices and various scenes in the video to be understood, and the prediction unit is specifically used for:

[0042] Based on the mapping relationship, the multimodal features of all slices in the target scene where the target slice is located are subjected to average pooling to obtain the target scene splicing features;

[0043] Based on the target scene splicing features, the relationships between people and / or between people and locations in the target scene are predicted to obtain relationship prediction results;

[0044] Based on the target multimodal features corresponding to the target slice and the target scene stitching features, the interactions between people and / or between people and locations in the target slice are predicted to obtain the interaction prediction results.

[0045] Thirdly, a deep video understanding device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the aforementioned deep video understanding method.

[0046] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned deep video understanding method.

[0047] The beneficial effects of the technical solution provided in this application include: effectively improving the accuracy of prediction results such as relationships and interactions.

[0048] This application provides a deep video understanding method, apparatus, device, and readable storage medium, including: acquiring text features corresponding to each slice in the video to be understood; extracting features from each slice to obtain overall slice features; performing face detection and tracking on people in each slice to generate a tracking sequence video, and extracting features from the tracking sequence video to obtain person tracking video features; determining the location type corresponding to each slice based on a residual network and the overall slice features; randomly combining people and locations in each slice based on the person tracking video features and the location type to obtain target features corresponding to each slice, wherein the target features include person-person pair features and person-location pair features; concatenating the text features, overall slice features, and target features corresponding to each slice to obtain multimodal features corresponding to each slice; and predicting the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features to obtain prediction results. This application introduces person-location pairs, i.e., adds location recognition, to enable more reasonable predictions of person interactions in different scenarios and locations. It also uses multimodal features formed by splicing text, slice overall features, person-person pair features, and person-location pair features to make more accurate predictions of interactions and relationships, thereby effectively improving the accuracy of prediction results such as relationships and interactions. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating a deep video understanding method provided in an embodiment of this application;

[0051] Figure 2 A schematic diagram illustrating the principle of forming multimodal features as provided in the embodiments of this application;

[0052] Figure 3 A schematic diagram illustrating the specific process of interaction and relationship prediction provided in an embodiment of this application;

[0053] Figure 4 A schematic diagram illustrating the principle of interaction and relationship prediction provided in the embodiments of this application;

[0054] Figure 5 This is a schematic diagram of the structure of a depth video understanding device provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of a depth video understanding device provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] This application provides a deep video understanding method, apparatus, device, and readable storage medium, which can solve the problem of low accuracy of prediction results caused by relying solely on visual features to predict relationships, interactions, etc. in related technologies.

[0058] See Figure 1 and Figure 2 As shown in the figure, this application provides a deep video understanding method, including the following steps:

[0059] Step S10: Obtain the text features corresponding to each slice in the video to be understood;

[0060] Exemplary and understandable, the core of video understanding algorithms is to understand the three elements of objects, actions, and scenes; among them, (1) regarding actor recognition and role relationship recognition: movies unfold stories with people at the core, so the recognition of actor identities is the foundation of video understanding; early identity recognition was accomplished through graph matching, by constructing face / name affinity matrices based on the number of times they appear in the video and the script respectively, after normalization, using the number of appearances as the weight of the edges, transforming the matrix into a graph, and transforming the problem into a matching problem between two affinity graphs, and then using spectral matching to match the vertices in the two graphs one by one to obtain face-name matching; subsequently, by constructing a graph library, extracting the features of faces in the video and performing similarity matching with the features of faces in the graph library, the identity matching problem was transformed into a graph library query problem.

[0061] In addition, the relationships between characters are also a key aspect of understanding the film. Since the relationships between characters are not an explicit visual feature, they can be identified by using features from different modalities or by selectively integrating existing features. For example, by integrating knowledge of social relations, features such as age, hairstyle, expression, and clothing can be used to assist in the identification of relationships.

[0062] (2) Regarding action and interaction recognition: Action recognition is mainly divided into two-stream network and 3D convolution method. The idea of ​​two-stream is to divide the network into spatial flow convolutional network and temporal flow convolutional network. After recognition is performed on the two networks respectively, the scores are fused to obtain the final recognition result. 3D convolution directly extracts features from the video by using 3D convolution kernels, without the need to model the temporal dimension separately.

[0063] In terms of interaction recognition, there are mainly two-stage and single-stage methods. The two-stage method divides the task into object detection and interaction judgment. The single-stage method mainly uses the midpoint of the detection box to represent people and objects, and the midpoint of two points to represent the interaction point. This model uses two parallel branches for detection and matching. The point prediction branch is used to predict people, objects and interaction points, and the matching branch is used to predict the displacement of the interaction point to its corresponding person and object point. Person points and object points from the same interaction point will be regarded as a matching pair, thereby predicting the final interaction.

[0064] (3) Regarding scene recognition: Scene recognition can use two combined deep networks to process images of different resolutions. The shallower network processes low-resolution images to obtain global information, while the deeper network processes high-resolution images to obtain more local information. The design idea of ​​combining global and local information makes the network have a more refined grasp of image information.

[0065] It should be understood that the actions of characters in many relationships in film and television works often overlap significantly. For example, two people eating, talking, or exercising together could be friends, parents and children, spouses, or siblings. Therefore, multimodal features become particularly important in such cases. For instance, dialogue containing prominent words like "dad," "boss," or "brother" can provide crucial information. Thus, this embodiment will acquire the text features corresponding to each segment of the video to be understood to assist in predicting the relationships between person-person pairs or person-location pairs.

[0066] However, data preparation is necessary before acquiring text features: First, a face database and location database can be selected based on the image set provided by TRECVID; then, the subtitles of the movie to be understood can be transcribed into text information using ASR (Automatic Speech Recognition) technology, and the movie to be understood can be segmented based on the subtitle text. The start and end times of each complete line of dialogue can be used as the segmentation criteria. Assuming that a scene in the movie to be understood is segmented into n slices, then the scene can be represented as V = {v1, v2…v}. nAfter saving the slices, a "movie-slice" and "scene-slice" mapping table is generated by combining the scene information. That is, the slices are respectively mapped to the movie to be understood and the scenes in the movie to be understood.

[0067] In this embodiment, the BERT-base model can be used to extract text features, that is, to obtain an n×768 feature matrix, where n is the number of slices and 768 is the feature dimension of a slice, and the matrix is ​​stored according to a predetermined structure.

[0068] Step S20: Extract features from each slice to obtain the overall features of each slice;

[0069] As an example, in this embodiment, the TSM video feature extraction model can be used to extract features from each slice, thereby obtaining the overall features of each slice, i.e., an n×2048 feature matrix.

[0070] Step S30: Perform face detection and tracking on the people in each slice to generate a tracking sequence video, and extract features from the tracking sequence video to obtain the people tracking video features;

[0071] As an example, it should be understood that this embodiment will obtain tracking sequences of different actors by performing face detection and tracking on the scene, calculating similarity with images in the face database, saving them as videos, and then extracting the video features of the character tracking.

[0072] Furthermore, the step of performing face detection and tracking on the individuals in each slice to generate a tracking sequence video includes:

[0073] Face detection is performed on the characters in each scene of the video to determine the identity information of each character;

[0074] Track the characters in each scene and generate multiple character tracking sequences;

[0075] Based on the identity information, multiple person tracking sequences are merged to obtain a target tracking sequence;

[0076] Based on the mapping relationship, the target tracking sequence, and the start and end times of the slice, a tracking sequence video corresponding to each slice is generated.

[0077] In this exemplary embodiment, face detection can be performed frame-by-frame in each scene of the video to be understood using the SCRFD model and the ArcFace model to extract facial features. The extracted facial features are then compared with facial features in a pre-defined face database to calculate similarity, and the similarity matrix is ​​saved to confirm facial identity, i.e., to determine the identity information of each person. Next, the Faster R-CNN model and the Deepsort algorithm are combined to track people in the scene, and the original tracking information is saved to obtain a series of tracking figures. Finally, the identities of people in the tracking sequence are predicted using the tracking sequence and the facial identities frame-by-frame, and different sequences with consistent identity information are merged into a single sequence.

[0078] For example, if the person in tracking sequence 1 is A, the person in tracking sequence 2 is B, and the person in tracking sequence 3 is A, then tracking sequence 1 and tracking sequence 3 are merged to obtain the merged tracking sequence. However, since the person in tracking sequence 2 is different from the person in other tracking sequences, there is no need to merge tracking sequence 2. Therefore, tracking sequence 2 and the merged tracking sequence are used as the target tracking sequence.

[0079] Then, based on the target tracking sequence and the start and end times of each video slice, a tracking sequence video corresponding to each slice is generated, and the features of the person tracking video are extracted by the TSM model, thus obtaining and saving an n×2048 feature matrix.

[0080] Step S40: Determine the location type corresponding to each slice based on the residual network and the overall features of the slice;

[0081] It should be understood, as an example, that most existing technologies, when using multimodal information for prediction, neglect contextual information, i.e., location information; however, location information is very helpful for interaction recognition. For example, in a restaurant, people are likely to engage in actions such as eating and talking; in a car, the action would be driving. Furthermore, in relationship prediction, specific locations can provide highly discriminative information. For instance, if the location is at home, the relationship between the people in that scene is likely to be that of relatives; if the location is in the office, then it is very likely to be a superior-subordinate or colleague relationship.

[0082] Therefore, this embodiment will extract location features from the overall features of the slice using ResNet, and calculate the similarity matrix between the location features and locations in the preset location database, thereby obtaining the location type corresponding to each location in the slice, so as to make more reasonable predictions of human interaction in different scene locations.

[0083] Step S50: Based on the video features of person tracking and the location type, randomly combine the people and locations in each slice to obtain the target features corresponding to each slice. The target features include person-person pair features and person-location pair features.

[0084] As an example, in this embodiment, the people and locations detected in each slice are randomly combined according to the characteristics of the person tracking video and the location type to obtain the person-person pair features and person-location pair features corresponding to each slice. For example, if person A, person B, and location C are detected in slice 1, then person A, person B, and location C can be randomly combined to obtain person A-person B pair, person A-location C pair, and person B-location C pair.

[0085] Step S60: Concatenate the text features, overall features of the slice, and target features corresponding to each slice to obtain the multimodal features corresponding to each slice;

[0086] As an example, in this embodiment, the text features, overall features of the slice, and person-person / person-location features extracted from each slice are concatenated. That is, all features are passed through two fully connected layers separately and then directly concatenated to obtain the multimodal feature vector (i.e., multimodal features) corresponding to each slice, and the feature dimension is 1536.

[0087] Step S70: Based on the multimodal features, predict the interactions and relationships between people and / or between people and locations in the video to be understood, and obtain the prediction results.

[0088] As an example, in this embodiment, multimodal features are input into the model to combine the multimodal features of all slices in the scene where a single slice is located to predict the relationship between people and people / locations in the scene and to assist in predicting interactions. That is, the multimodal features of all slices in the scene where a single slice is located are used to assist the multimodal features of the single slice to predict the interactions between people-people pairs / people-location pairs, thereby improving the accuracy of prediction results such as relationships and interactions.

[0089] Furthermore, there is a mapping relationship between the slices and various scenes in the video to be understood, see [link to relevant documentation]. Figure 3 As shown, step S70 specifically includes:

[0090] Step S701: According to the mapping relationship, perform average pooling on the multimodal features of all slices in the target scene where the target slice is located to obtain the target scene splicing features;

[0091] Step S702: Based on the target scene stitching features, predict the relationships between people and / or between people and locations in the target scene to obtain relationship prediction results;

[0092] Specifically, the step of predicting the relationships between people and / or between people and locations in the target scene based on the target scene stitching features, to obtain relationship prediction results, includes:

[0093] Substitute the splicing features of the target scene into the first calculation formula to obtain the score of the relationship between people or between people and locations in the target scene.

[0094] The target relationship corresponding to the highest relationship score is taken as the relationship prediction result for the target scenario.

[0095] The first calculation formula is:

[0096]

[0097] In the formula, s R (V, r) represents the relation score of person-person pair p or person-location pair p for relation r in the target scene, where V represents the slice set corresponding to the target scene, Φ R This represents the splicing features of the target scene. and Both represent network parameters, σ r This represents the activation function.

[0098] Step S703: Based on the target multimodal features corresponding to the target slice and the target scene stitching features, predict the interactions between people and / or between people and locations in the target slice to obtain the interaction prediction results.

[0099] Specifically, the step of predicting the interactions between people and / or between people and locations in the target slice based on the target multimodal features corresponding to the target slice and the target scene stitching features, to obtain the interaction prediction result, includes:

[0100] Substitute the target multimodal features and target scene stitching features corresponding to the target slice into the second calculation formula to obtain the interaction scores between people or between people and locations in the target slice.

[0101] The interaction relationship corresponding to the highest interaction score is used as the interaction prediction result of the target slice;

[0102] The second calculation formula is:

[0103]

[0104] In the formula, s I (v, V, a) represents the interaction score of person-person pair p or person-location pair p on target slice v for interaction a, Φ I Φ represents the target's multimodal features. RThis represents the splicing features of the target scene. W I1 B I1 and Both represent network parameters, σ a This represents the activation function.

[0105] Exemplary, see Figure 4 As shown, taking the prediction of the interaction relationship between a target slice f and its target scene F as an example: after obtaining the multimodal features Φ corresponding to each slice in the target scene F, the multimodal features Φ of all slices in the target scene F are input into the prediction model for average pooling processing (i.e., Figure 4 The avg pool is used to obtain the target scene stitching feature Φ corresponding to the target scene F. R At this point, the target scene stitching feature Φ R The output will be split into two paths: one path will be fed into the classifier for relation prediction, and the other path will be fed into the interaction prediction channel to assist in the prediction of the target multimodal features Φ corresponding to the target slice f. I Conduct interactive predictions.

[0106] Among them, the target scene splicing feature Φ R After being input into the classifier, the relationship scores between people or between people and locations in the target scene F will be obtained using the following formula:

[0107]

[0108] In the formula, s R (V, r) represents the relation score of person-person pair p or person-location pair p for relation r in the target scene, where V represents the slice set corresponding to the target scene, Φ R This represents the splicing features of the target scene. and All represent network parameters. and The specific value of σ can be determined according to actual needs. r This represents the activation function, which is preferably the sigmoid activation function.

[0109] The relationship with the highest score is then used as the relationship prediction result for the target scenario F. For example, in the target scenario F, the relationship between character D and character E is friend with a score of 0.72, stranger with a score of 0.33, and manager with a score of 0.27. Since friend has the highest score, the relationship between character D and character E is set as friend as the relationship prediction result.

[0110] splice the target scene features Φ RAfter being input into the interactive prediction channel, it will be compared with the target multimodal features Φ corresponding to the target slice f. I The data are concatenated and fused together in a fully connected layer (fc) to obtain fused features. After the fused features are input into the classifier through the ReLU activation function, the interaction scores between people or between people and locations in the target slice f are obtained using the following formula:

[0111]

[0112] In the formula, s I (v, V, a) represents the interaction score of person-person pair p or person-location pair p on target slice v for interaction a, Φ I Φ represents the target's multimodal features. R This represents the splicing features of the target scene. W I1 B I1 and All represent network parameters. W I1 B I1 and The specific value of σ can be determined according to actual needs. a This represents the activation function, which is preferably the sigmoid activation function.

[0113] Finally, the interaction relationship with the highest score is used as the interaction prediction result for the target slice f. For example, in the target slice f, the interaction between person D and person E at location G has the following scores: asks score is 0.69, admits score is 0.65, watches score is 0.64, explains score is 0.35, and talks to score is 0.27. Since asks has the highest score, the interaction between person D and person E at location G is set as asks as the interaction prediction result.

[0114] In summary, since relationships often influence the interaction patterns of a pair of objects, this embodiment uses relationship prediction to assist in interaction prediction, thereby improving the accuracy of interaction prediction. Simultaneously, this embodiment introduces person-location pairs, i.e., adds location recognition, enabling more reasonable predictions of person interactions in different scene locations. Furthermore, this embodiment uses text, slice-based overall features, and person-person and person-location pair features to form a multimodal feature data that more completely represents the film. This multimodal feature data is then used for more accurate prediction of interactions and relationships, effectively improving the accuracy of relationship and interaction prediction results, and thus enhancing the accuracy and effectiveness of story film understanding.

[0115] Furthermore, after the step of predicting the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, the method further includes:

[0116] A knowledge graph corresponding to the video to be understood is generated based on the prediction results.

[0117] As an example, in this embodiment, the prediction results are imported into the database to generate a movie-level and scene-level knowledge graph corresponding to the movie to be understood; after the knowledge graph is generated, the movie-related query questions are parsed, converted into database query statements, the closest relationship is found in the database, and the answer is selected according to the integral algorithm to complete the retrieval.

[0118] Therefore, this embodiment proposes a method for predicting character interactions and relationships in a story film using multimodal information. Specifically, it obtains a complete feature representation of a film slice by splicing together the overall video features, dialogue text features, and character / location features. Then, it predicts the relationship between a pair of objects in a scene by combining the features of multiple slices, and uses this relationship to help predict the interaction between the pair of objects in a single slice. In other words, it automatically analyzes the film story content by fusing multiple information modalities such as text and video to form features, and automatically answers questions about short-term interactions and long-term relationships between characters in the film.

[0119] See Figure 5 As shown in the figure, this application embodiment also provides a deep video understanding device, including a feature extraction unit, a combination unit, a stitching unit and a prediction unit;

[0120] The feature extraction unit is used to obtain the text features corresponding to each slice in the video to be understood; to extract features from each slice to obtain the overall features of each slice; to perform face detection and tracking on the people in each slice to generate a tracking sequence video, and to extract features from the tracking sequence video to obtain the people tracking video features; and to determine the location type corresponding to each slice based on the residual network and the overall features of the slice.

[0121] The combination unit is used to randomly combine people and locations in each slice based on the video features of person tracking and the location type to obtain the target features corresponding to each slice. The target features include person-person pair features and person-location pair features.

[0122] The splicing unit is used to splice the text features, overall features of the slice, and target features corresponding to each slice to obtain the multimodal features corresponding to each slice.

[0123] The prediction unit is used to predict the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, and to obtain the prediction results.

[0124] Furthermore, there is a mapping relationship between the slices and various scenes in the video to be understood, and the prediction unit is specifically used for:

[0125] Based on the mapping relationship, the multimodal features of all slices in the target scene where the target slice is located are subjected to average pooling to obtain the target scene splicing features;

[0126] Based on the target scene splicing features, the relationships between people and / or between people and locations in the target scene are predicted to obtain relationship prediction results;

[0127] Based on the target multimodal features corresponding to the target slice and the target scene stitching features, the interactions between people and / or between people and locations in the target slice are predicted to obtain the interaction prediction results.

[0128] Furthermore, the prediction unit is specifically used for:

[0129] Substitute the splicing features of the target scene into the first calculation formula to obtain the score of the relationship between people or between people and locations in the target scene.

[0130] The target relationship corresponding to the highest relationship score is taken as the relationship prediction result for the target scenario.

[0131] The first calculation formula is:

[0132]

[0133] In the formula, s R (V, r) represents the relation score of person-person pair p or person-location pair p for relation r in the target scene, where V represents the slice set corresponding to the target scene, Φ R This represents the splicing features of the target scene. and Both represent network parameters, σ r This represents the activation function.

[0134] Furthermore, the prediction unit is specifically used for:

[0135] Substitute the target multimodal features and target scene stitching features corresponding to the target slice into the second calculation formula to obtain the interaction scores between people or between people and locations in the target slice.

[0136] The interaction relationship corresponding to the highest interaction score is used as the interaction prediction result of the target slice;

[0137] The second calculation formula is:

[0138]

[0139] In the formula, s I (v, V, a) represents the interaction score of person-person pair p or person-location pair p on target slice v for interaction a, Φ I Φ represents the target's multimodal features. R This represents the splicing features of the target scene. W I1 B I1 and Both represent network parameters, σ a This represents the activation function.

[0140] Furthermore, the feature extraction unit is specifically used for:

[0141] Face detection is performed on the characters in each scene of the video to determine the identity information of each character;

[0142] Track the characters in each scene and generate multiple character tracking sequences;

[0143] Based on the identity information, multiple person tracking sequences are merged to obtain a target tracking sequence;

[0144] Based on the mapping relationship, the target tracking sequence, and the start and end times of the slice, a tracking sequence video corresponding to each slice is generated.

[0145] Furthermore, the device also includes a graph generation unit, which is used to generate a knowledge graph corresponding to the video to be understood based on the prediction results.

[0146] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and each unit described above can be referred to the corresponding process in the aforementioned deep video understanding method embodiments, and will not be repeated here.

[0147] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 6 It runs on the depth video understanding device shown.

[0148] This application also provides a deep video understanding device, including: a memory, a processor, and a network interface connected via a system bus. The memory stores at least one instruction, which is loaded and executed by the processor to implement all or part of the steps of the aforementioned deep video understanding method.

[0149] The network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0150] A processor can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.

[0151] Memory can be used to store computer programs and / or modules. The processor performs various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as video playback, image playback, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as video data, image data, etc.). Furthermore, memory can include high-speed random access memory (RAM), and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital Cards (SD cards), Flash Cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0152] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements all or part of the steps of the aforementioned deep video understanding method.

[0153] The embodiments of this application can implement all or part of the aforementioned processes, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various methods described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, servers, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0157] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A deep video understanding method, characterized in that, Includes the following steps: Obtain the text features corresponding to each slice in the video to be understood; Feature extraction is performed on each slice to obtain the overall slice features corresponding to each slice; Face detection and tracking are performed on the people in each slice to generate tracking sequence videos. Feature extraction is then performed on the tracking sequence videos to obtain the features of the people tracking videos. The location type corresponding to each slice is determined based on the residual network and the overall characteristics of the slice; Based on the video features of people tracking and the location type, people and locations in each slice are randomly combined to obtain the target features corresponding to each slice. The target features include people-people pair features and people-location pair features. The text features, overall features of each slice, and target features corresponding to each slice are concatenated to obtain the multimodal features corresponding to each slice. Based on the multimodal features, predictions are made regarding the interactions and relationships between people and / or between people and locations in the video to be understood, and prediction results are obtained. The slices are mapped to various scenes in the video to be understood. The prediction of interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, to obtain prediction results, includes: Based on the mapping relationship, the multimodal features of all slices in the target scene where the target slice is located are subjected to average pooling to obtain the target scene splicing features; Based on the target scene splicing features, the relationships between people and / or between people and locations in the target scene are predicted to obtain relationship prediction results; Based on the target multimodal features corresponding to the target slice and the target scene stitching features, the interaction between people and / or between people and places in the target slice is predicted to obtain the interaction prediction result; The prediction of relationships between people and / or between people and locations in the target scene based on the target scene stitching features, to obtain relationship prediction results, includes: Substitute the splicing features of the target scene into the first calculation formula to obtain the score of the relationship between people or between people and locations in the target scene. The target relationship corresponding to the highest relationship score is taken as the relationship prediction result for the target scenario. The first calculation formula is: In the formula, Character-character pair Or person-location pair Relationships in the target scenario Relationship score, This represents the slice set corresponding to the target scene. This represents the splicing features of the target scene. and All represent network parameters. This represents the activation function.

2. The depth video understanding method as described in claim 1, characterized in that, The step of predicting the interactions between people and / or between people and locations in the target slice based on the target multimodal features corresponding to the target slice and the target scene stitching features, to obtain the interaction prediction results, includes: Substitute the target multimodal features and target scene stitching features corresponding to the target slice into the second calculation formula to obtain the interaction scores between people or between people and locations in the target slice. The interaction relationship corresponding to the highest interaction score is used as the interaction prediction result of the target slice; The second calculation formula is: In the formula, Character-character pair Or person-location pair In the target slice For interaction Interaction score, Represents the multimodal features of the target. This represents the splicing features of the target scene. All represent network parameters. This represents the activation function.

3. The depth video understanding method as described in claim 1, characterized in that, The process of performing face detection and tracking on the individuals in each slice to generate a tracking sequence video includes: Face detection is performed on the characters in each scene of the video to determine the identity information of each character; Track the characters in each scene and generate multiple character tracking sequences; Based on the identity information, multiple person tracking sequences are merged to obtain a target tracking sequence; Based on the mapping relationship, the target tracking sequence, and the start and end times of the slice, a tracking sequence video corresponding to each slice is generated.

4. The depth video understanding method as described in claim 1, characterized in that, After the step of predicting the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, the method further includes: A knowledge graph corresponding to the video to be understood is generated based on the prediction results.

5. A depth video understanding device, characterized in that: It includes feature extraction units, combination units, splicing units, and prediction units; The feature extraction unit is used to obtain the text features corresponding to each slice in the video to be understood; Feature extraction is performed on each slice to obtain the overall slice features corresponding to each slice; Face detection and tracking are performed on the people in each slice to generate a tracking sequence video. Feature extraction is performed on the tracking sequence video to obtain the person tracking video features. The location type corresponding to each slice is determined based on the residual network and the overall features of the slice. The combination unit is used to randomly combine people and locations in each slice based on the video features of person tracking and the location type to obtain the target features corresponding to each slice. The target features include person-person pair features and person-location pair features. The splicing unit is used to splice the text features, overall features of the slice, and target features corresponding to each slice to obtain the multimodal features corresponding to each slice. The prediction unit is used to predict the interactions and relationships between people and / or between people and locations in the video to be understood based on the multimodal features, and to obtain the prediction results. The slices are mapped to various scenes in the video to be understood, and the prediction unit is specifically used for: Based on the mapping relationship, the multimodal features of all slices in the target scene where the target slice is located are subjected to average pooling to obtain the target scene splicing features; Based on the target scene splicing features, the relationships between people and / or between people and locations in the target scene are predicted to obtain relationship prediction results; Based on the target multimodal features corresponding to the target slice and the target scene stitching features, the interaction between people and / or between people and places in the target slice is predicted to obtain the interaction prediction result; The prediction unit is further specifically used for: Substitute the splicing features of the target scene into the first calculation formula to obtain the score of the relationship between people or between people and locations in the target scene. The target relationship corresponding to the highest relationship score is taken as the relationship prediction result for the target scenario. The first calculation formula is: In the formula, Character-character pair Or person-location pair Relationships in the target scenario Relationship score, This represents the slice set corresponding to the target scene. This represents the splicing features of the target scene. and All represent network parameters. This represents the activation function.

6. A depth video understanding device, characterized in that, include: A memory and a processor, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the depth video understanding method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the depth video understanding method according to any one of claims 1 to 4.

Images