A method, device and related product for capturing evolution of a person's social relationship

Abstract

The application discloses a method, device and related product for capturing evolution of a person's social relationship. The method comprises: first processing a target video into a video clip frame sequence, and extracting multi-view visual features of each video clip frame in the video clip frame sequence according to a preset algorithm to generate a clip-level graph. Then, the clip-level graph is processed based on a first preset attention mechanism to determine scene-level person features and person pair features. The scene-level social relationship graph of the person role is determined by using the features. In this way, the scene-level person features and person pair features obtained through the video clip frame sequence, the preset algorithm and the first preset attention mechanism processing contain multi-view and multi-modal information, and the features contain changes in the time dimension. Thus, the technical problem that the prior art only considers the identification of the person relationship in a short video and ignores the time evolution factor is overcome, and the accuracy of capturing the evolution of the social relationship is improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and natural language processing, and in particular to a method, apparatus and related products for capturing the evolution of human social relationships. Background Technology

[0002] As online video media platforms offer increasingly intelligent services, such as semantic-driven video retrieval and recommendation, it is becoming increasingly important to study the social relationships between people in videos in order to obtain a better viewing experience.

[0003] Existing techniques for studying social relationships between people in videos aim to identify these relationships in short videos. However, for complete videos, due to the longer input length, existing techniques struggle to summarize multimodal cues into a holistic semantic representation of the entire video and do not consider temporal evolution factors. This results in a failure to fully summarize the evolutionary cues of social relationships, leading to low accuracy in capturing these relationships. Summary of the Invention

[0004] In view of this, the embodiments of this application provide a method, apparatus and related products for capturing the evolution of a person's social relationships, aiming to fully summarize the evolutionary clues of a person's social relationships, thereby improving the accuracy of capturing the evolutionary relationships of a person's social relationships.

[0005] In a first aspect, embodiments of this application provide a method for capturing the evolution of a person's social relationships, the method comprising:

[0006] In response to the acquired target video, a sequence of video clip frames with related relationships is generated; the sequence of video clip frames includes characters.

[0007] According to a preset algorithm, multi-view visual features of each video clip frame in the video clip frame sequence are extracted to construct a clip-level graph for each video clip; the clip-level graph is used to represent the state of a person's social relationship.

[0008] The clip-level image is processed based on a first preset attention mechanism to determine scene-level character features and character pair features; wherein, the first preset attention mechanism is to include historical information evolution trends of each time period, so that the scene-level character features and character pair features include time dimension information.

[0009] Based on the scene-level character features and character pair features, a scene-level social relationship graph of the character roles is determined to capture the evolution of character social relationships in the target video.

[0010] Optionally, the step of extracting multi-view visual features of each video clip frame in the video clip frame sequence according to a preset algorithm, and constructing a clip-level image for each video clip, includes:

[0011] Based on a first preset algorithm, character features, character pair features, and segment video features are extracted from the video clip frame sequence; the segment video features include scene information.

[0012] Based on the second preset algorithm, dialogue features are extracted from the dialogue files in the video clip frame sequence; wherein, the character features, character pair features, segment video features and dialogue features constitute the multi-view visual features;

[0013] Based on the multi-view visual features and the second preset attention mechanism, a clip-level graph is constructed for each video clip; the clip-level graph is composed of the multi-view visual features; the second preset attention mechanism is a graph attention mechanism used to construct the clip-level graph.

[0014] Optionally, constructing a clip-level graph for each video clip based on the multi-view visual features and the second preset attention mechanism includes:

[0015] The multi-view visual features are used as different nodes of the clip-level graph, and the information of the different nodes is propagated using a second preset attention mechanism.

[0016] Different weights are assigned to the different nodes, and information is passed to adjacent nodes to generate a clip-level graph for each video clip.

[0017] Optionally, determining the scene-level social relationship graph of the character based on the scene-level character features and character pair features includes:

[0018] The scene-level character features and character pair features are input into a preset third preset attention mechanism to adjust the scene-level character pair features;

[0019] The adjusted scene-level character pair features are sent to the fully connected layer to classify the social relationships of the character pairs and obtain the scene-level social relationship graph of the characters.

[0020] Optionally, the method further includes:

[0021] Based on the determined scene-level social relationships and time-dimensional sliding window of the characters, scene-level social relationship graphs of the characters in different video clip frame sequences are generated.

[0022] The scene-level social relationship graphs of the aforementioned characters are merged according to the chronological evolution order to generate a global evolutionary social relationship graph.

[0023] Optionally, in response to the acquired target video, generating a sequence of video clip frames with related relationships includes:

[0024] Acquire the target video;

[0025] The target video is divided into multiple video clips based on the plot and scene;

[0026] The multiple video clips are sampled at equal intervals to obtain a video clip frame sequence;

[0027] The video clip frame sequence is subjected to character localization and re-identification to determine the correlation between video clip frames and generate a video clip frame sequence with correlation.

[0028] Optionally, the step of performing character detection and re-identification on the video clip frame sequence to determine the association between video clip frames includes:

[0029] The video clip frames are detected based on a pre-trained person detector, and the person bounding boxes and their corresponding confidence scores are obtained.

[0030] In response to the confidence level being greater than a first preset threshold, it is determined that the character frame contains a character.

[0031] The face bounding boxes marked with identifiers are matched with the character bounding boxes containing the character roles, and the character bounding boxes are re-identified;

[0032] Based on the third preset algorithm, the volume features of all character frames containing the character are extracted to determine the character frame features;

[0033] The similarity between the person bounding box features and the labeled person bounding box features is calculated based on cosine similarity.

[0034] If the similarity is greater than a preset threshold, the character frame and the labeled character frame are the same character.

[0035] Secondly, embodiments of this application provide an apparatus for capturing the evolution of a person's social relationships, the apparatus comprising:

[0036] A response unit is used to generate a sequence of video clip frames with related relationships in response to the acquired target video; the sequence of video clip frames includes characters.

[0037] The clip-level graph generation unit is used to extract multi-view visual features of each video clip frame in the video clip frame sequence according to a preset algorithm, and construct a clip-level graph for each video clip; the clip-level graph is used to represent the state of social relationships of people.

[0038] The determining unit is used to process the clip-level image based on a first preset attention mechanism to determine scene-level character features and character pair features; wherein, the first preset attention mechanism is to include historical information evolution trends of each time period, and is used to make the scene-level character features and character pair features include time dimension information;

[0039] The capture unit is used to determine the scene-level social relationship graph of the characters based on the scene-level character features and character pair features, so as to capture the evolution of the characters' social relationships in the target video.

[0040] Thirdly, embodiments of this application provide a generation device for a method of capturing the evolution of social relationships of individuals, the device comprising a processor and a memory:

[0041] The memory is used to store program code and transmit the program code to the processor;

[0042] The processor is configured to execute, according to instructions in the program code, the steps of the method for capturing the evolution of human social relationships as described in any of the preceding first aspects.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program or code that, when executed by a processor, implements the steps of the method for capturing the evolution of human social relationships as described in the first aspect above.

[0044] This application discloses a method, apparatus, and related products for capturing the evolution of social relationships between people in videos. When executing the method, firstly, in response to the acquired target video, a sequence of video clip frames with associated relationships is generated. Then, according to a preset algorithm, multi-view features of each video clip frame in the video clip frame sequence are extracted to generate a clip-level graph. Next, the clip-level graph is processed based on a first preset attention mechanism to determine scene-level person features and person-pair features. The scene-level person features and person-pair features are then used to determine the scene-level social relationship graph of the characters. Thus, by processing the video clip frame sequence in conjunction with the preset algorithm and the first preset attention mechanism, the acquired scene-level person features and person-pair features contain multi-view and multi-modal information, and the features capture changes in the temporal dimension. This overcomes the technical problem of existing technologies that only consider the identification of person relationships in short videos while ignoring temporal evolution factors, and improves the accuracy of capturing the evolution of social relationships. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a method for capturing the evolution of character relationships provided in this application embodiment;

[0047] Figure 2 A flowchart illustrating another method for capturing the evolution of interpersonal relationships provided in this application embodiment;

[0048] Figure 3 A flowchart illustrating a third method for capturing the evolution of interpersonal relationships provided in this application embodiment;

[0049] Figure 4 This is an example diagram illustrating the process of obtaining globally evolving social relationships from video clip frame sequences provided in this application embodiment.

[0050] Figure 5 A schematic diagram of the device structure for capturing the third type of character relationship evolution provided in this application embodiment. Detailed Implementation

[0051] As mentioned earlier, existing technologies for studying social relationships between people in videos aim to identify these relationships in short videos. However, for complete videos, the video length is much longer. Existing technologies struggle to summarize multimodal cues into a holistic semantic representation of the relationships between people throughout the entire video, and they do not consider temporal evolution factors. This makes it difficult for existing technologies to fully summarize the evolutionary cues of people's social relationships, resulting in low accuracy in capturing these evolving social relationships.

[0052] Based on this, this application proposes a method for capturing the evolution of character relationships. This method generates a sequence of video clips comprising multiple video frames through the entire video processing. The sequence is then processed using a preset algorithm and a first preset attention mechanism to generate scene-level character features and character pair features with multi-perspective information and temporal evolution relationships. By utilizing the acquired scene-level character features and character pair features to generate a scene-level social relationship graph between characters, the social evolution relationships of characters can be accurately captured.

[0053] To enable those skilled in the art to better understand the present application, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0054] See Figure 1 The flowchart below illustrates a method for capturing the evolution of interpersonal relationships, provided in an embodiment of this application. This method can be applied to video analysis systems. The method includes at least the following steps:

[0055] S101: In response to the acquired target video, generate a sequence of video clip frames with related relationships.

[0056] In this application embodiment, the target video can be film or television content for which character analysis is required. Specifically, the target video can be obtained from online video media through video editing or other methods. In this application embodiment, the target video can be a long video.

[0057] In this embodiment, after acquiring the target video, the video analysis system needs to process the target video to generate a video clip frame sequence with multiple video clip frames. In this embodiment, the video clip frame sequence includes characters and relationships between each video clip frame.

[0058] In this embodiment, to ensure content independence and integrity, and to ensure the correlation between each video clip frame, segmentation and sampling can be used, along with character localization and re-identification. Specific implementation methods are detailed below. Figure 2 A detailed description will be provided in that section. This will not be discussed further here.

[0059] S102: According to the preset algorithm, extract the multi-view visual features of each video clip frame in the video clip frame sequence, and construct the clip-level map of each video clip.

[0060] For the generated sequence of video clip frames, multi-view visual features of each video clip frame can be extracted using a preset algorithm. Then, based on the multi-view visual features, a clip-level graph for each video clip frame is constructed. The clip-level graph is used to structurally represent the social state of the video clip frames. In this embodiment, the multi-view visual features include character features, character pair features, dialogue features, and segment video features. The segment video features are used to represent the scene information contained in the segment.

[0061] In this embodiment, multi-view visual features are extracted according to a preset algorithm, which can represent characters and character pairs based on the visual features of character bounding boxes and union boxes. In this embodiment, character pairs represent the relationship between characters. Specifically:

[0062] Based on a first preset algorithm, character features, character pair features, and segment video features are extracted from the video clip frame sequence. Dialogue features are extracted from the dialogue files in the video clip frame sequence based on a second preset algorithm. A clip-level graph is constructed for each video clip based on multi-view visual features and a second preset attention mechanism. The clip-level graph is composed of the multi-view visual features. In this embodiment, the second preset attention mechanism is a graph attention mechanism used to construct the clip-level graph.

[0063] Optionally, multi-view visual features can be used as different nodes in the clip-level graph, and information from these different nodes can be propagated using a second preset attention mechanism. Different weights are assigned to different nodes, and information is passed to adjacent nodes to generate a clip-level graph for each video clip. In this embodiment, the second preset attention mechanism can be a graph attention network mechanism.

[0064] Exemplary illustration:

[0065] In this embodiment, a dilated 3D convolutional residual network pre-trained on the Kinetics-400 dataset can be used to process person bounding boxes and person joint bounding boxes, extracting person features and person pair features. The video clip frame sequence is input into a pre-trained dilated 3D convolutional residual network to extract video segment features. Deep self-attention transform networks extract dialogue features from the dialogue file of the current clip. These four different features constitute the four types of nodes on the current clip-level graph.

[0066] The system aggregates information from the edited graph and uses a graph attention network to propagate node information. In this embodiment, different weights can be assigned to different nodes, and information can be passed between adjacent node pairs. This enhances the features of individuals and their pairs. Specifically, given a graph node embedding of the same dimension, operations on video edited frames can be described as follows:

[0067]

[0068]

[0069] in, Represents any node in the clip-level graph. The representative model's first In the layer, H represents the number of attention heads in the multi-headed attention of the model, exp represents the exponential function, and W is a learnable matrix. These are the learnable attention weights between nodes i and j. It is the set of nearest neighbors of node i in the current clipping level graph. The weight vector of the attention function implemented in the feedforward layer; yes Activation function. This represents a join operation. Where... The connection process is based on the adjacency matrix. This was facilitated. The specific definitions in the clip-level diagram are as follows:

[0070]

[0071] Finally, multi-view visual features are used to obtain clip-level images via a graph attention network. This is used to enhance characters and character pairs in video clip frames. The specific enhancement formula is as follows:

[0072]

[0073] GAT stands for Graph Attention Network.

[0074] S103: Process the clip-level image based on the first preset attention mechanism to determine scene-level character features and character pair features.

[0075] The enhanced representations of character features and character pair features in the determined edit-level images are input into a first preset attention mechanism for processing. In this embodiment, the first preset attention mechanism includes historical information evolution trends for each time period, enabling the scene-level character features and character pair features to incorporate temporal dimension information. That is, the first preset attention mechanism can aggregate the evolution trends of historical information for each time period and capture time dependencies. This ensures that the acquired scene-level character features and character pair features include multi-view information and exhibit temporal changes.

[0076] S104: Based on scene-level character features and character pair features, determine the scene-level social relationship graph of the character roles in order to capture the evolution of character social relationships in the target video.

[0077] Based on scene-level character features and character pair features, a scene-level social relationship graph for each character is determined. In this embodiment, the scene-level character features and character pair features can be input into a preset third attention mechanism to adjust the scene-level character pair features. The adjusted scene-level character pair features are then sent to a fully connected layer to classify the social relationships of the character pairs, thereby obtaining the scene-level social relationship graph for each character.

[0078] Example illustration: Assuming scene-level character features and the characteristics of the characters Scene-level character features and the characteristics of the characters Input to a scene-level graph attention network to add scene-level character features and the characteristics of the characters At this time, the characteristics of the characters and the characteristics of the characters It contains information from multiple perspectives and exhibits changes over time. It also includes multimodal information.

[0079]

[0080] The scene-level graph critical matrix A is:

[0081]

[0082] Finally, the enhanced character features The data is fed into a fully connected layer, and the social relationships of the corresponding pairs of people are classified in a supervised manner to obtain a scene-level social relationship graph between people.

[0083] This application discloses a method for capturing the evolution of social relationships among people in videos. First, in response to the acquired target video, a sequence of video clip frames with related relationships is generated. Then, according to a preset algorithm, multi-view features of each video clip frame in the sequence are extracted to generate a clip-level graph. Next, the clip-level graph is processed based on a first preset attention mechanism to determine scene-level person features and person-pair features. The scene-level person features and person-pair features are then used to determine the scene-level social relationship graph of the characters. Thus, by using a video clip frame sequence, combined with the preset algorithm and the first preset attention mechanism, the acquired scene-level person features and person-pair features contain multi-view and multi-modal information, and the features capture changes in the temporal dimension. This overcomes the technical problem of existing technologies that only consider person relationship recognition while ignoring temporal evolution factors, and improves the accuracy of capturing the evolution of social relationships.

[0084] See Figure 2 This is a flowchart illustrating another method for capturing the evolution of character relationships provided in this application. This method, when generating a video clip frame sequence, fully considers content independence and semantic integrity, as well as the relationships between segments, thereby further improving the accuracy of capturing character relationships using this flowchart. The method is illustrated using a movie dataset as an example. The method includes at least the following steps:

[0085] S201: Obtain the target video.

[0086] S202: Divide the target video into multiple video clips based on the plot and scene.

[0087] In this embodiment of the application, the complete video is divided into multiple time-edited segments according to the plot and scene, which can ensure the independence of the content and the integrity of the video.

[0088] S203: Sample multiple video clips at equal intervals to obtain a video clip frame sequence.

[0089] By sampling the multiple video clips acquired in S202 at equal intervals, a video clip frame sequence can be obtained. Example: Each video clip is sampled at a sampling frequency of 1 frame / second to obtain multiple video clip frames. The sequence composed of these video clips is the video clip frame sequence.

[0090] S204: Perform character localization and re-identification on the video clip frame sequence to generate a video clip frame sequence with related relationships.

[0091] In this embodiment, character localization and re-identification can be performed on the video clip frame sequence to establish the association between each video clip frame. This allows the video clip frame sequence to have a hypothetical association between characters.

[0092] In this application embodiment, character localization and re-identification can be achieved in the following ways:

[0093] A pre-trained person detector detects video clip frames, obtaining person bounding boxes and their corresponding confidence scores. When the confidence score of a person bounding box is greater than a first preset threshold, it is determined that the person bounding box contains a person. Next, face bounding boxes with identifiers are used to match the person bounding boxes containing the person, performing re-identification. Volumetric features are extracted from all person bounding boxes containing the person based on a third preset algorithm to determine the person bounding box features. Finally, the similarity between the person bounding box features and the labeled person bounding box features is calculated based on cosine similarity. When the similarity is greater than a preset threshold, the person bounding box and the labeled person bounding box belong to the same person, completing the localization of the person.

[0094] S205: According to the preset algorithm, extract the multi-view visual features of each video clip frame in the video clip frame sequence, and construct the clip-level map of each video clip.

[0095] S206: Processing clip-level images based on temporal self-attention mechanism to determine scene-level character features and character pair features.

[0096] S207: Input scene-level character features and character pair features into the scene-level graph attention network to enhance scene-level character features and character pair features.

[0097] S208: Input the features of the person pairs into the fully connected layer, classify the social relationships of the corresponding person pairs through a supervised approach, and generate a scene-level social relationship graph between people.

[0098] S205~S208 and Figure 1 S102 to S104 are the same. They will not be discussed further here.

[0099] See Figure 3 This is a flowchart illustrating a third method for capturing the evolution of character relationships provided in this application. This method can generate a global evolutionary social relationship graph and visualize the evolution of relationships. The method is illustrated using a specific movie dataset as an example. The method includes at least the following steps:

[0100] S301: Sample the complete video according to the plot, scene and equal intervals to obtain a video clip frame sequence.

[0101] In this embodiment, the complete video is derived from a movie dataset on an online video platform. Equal-interval sampling involves sampling at a frequency of 1 frame per second to obtain a sequence of video clip frames composed of video clips.

[0102] S302: Locate and re-identify characters in the video editing frame sequence.

[0103] In this embodiment, a Faster R-CNN-based person detector can be used to detect people in video clip frames, selecting detection boxes with a confidence score greater than 0.8 as the detected person bounding boxes. After obtaining the person bounding boxes, face bounding boxes with ID tags are used for overlap matching, and some person bounding boxes are re-identified. Then, a feature extraction network based on a residual network is used to extract features from all person bounding boxes, and cosine similarity is calculated between the features of the person bounding boxes. Unlabeled person bounding boxes with a similarity score greater than the labeled person bounding box threshold of 0.8 are considered to be the same person.

[0104] S303: Take the video clip frames in the video clip frame sequence and construct a clip-level graph through a graph attention network mechanism to enhance the features of the characters and their pairs.

[0105] S304: Input the enhanced character features and character pair features into the temporal self-attention network to generate scene-level character features and character pair features.

[0106] S305: Input scene-level character features and character pair features into the scene-level graph convolutional network to further enhance scene-level character features and character pair features.

[0107] S306: Input the enhanced scene-level character pairs into the fully connected layer to determine the scene-level social relationship graph between characters. .

[0108] S303~S306 and Figure 1 S102 to S104 are the same. They will not be discussed further here.

[0109] S307: According to the established By sliding a window along the time dimension, scene-level social relationship graphs of characters in different video clip frame sequences can be generated.

[0110] S308: Merge scene-level social relationship graphs of characters according to the time evolution order to generate a global evolutionary social relationship graph.

[0111] S309: Visualize the scene-level social relationship graph between characters and the global evolutionary social relationship graph.

[0112] Exemplary illustration:

[0113] See Figure 4 This is an example diagram illustrating the process of obtaining global evolutionary social relationships from a video clip frame sequence provided in this application embodiment. The video clip sequence (…) Each video clip can form a graph attention network, and these graph attention networks form clip-level graphs. The V and D distributions are used to represent the features of the video clips. Peaceful characteristics The nodes are represented by C and P distributions, which represent the nodes corresponding to character features and their pairs. The clip-level graph is input into a temporal self-attention network and a scene-level graph attention network. After relationship classification, scene-level social relationships are obtained. These scene-level social relationships are then slid along a time-dimensional window to generate globally evolving social relationships.

[0114] This application also provides an apparatus for capturing the evolution of character relationships. See also... Figure 5 This is a schematic diagram of a device for capturing the evolution of character relationships, provided in an embodiment of this application. The device 500 includes at least the following parts:

[0115] The response unit 501 is used to generate a sequence of video clip frames with related relationships in response to the acquired target video. The video clip frame sequence includes character roles.

[0116] The clip-level graph generation unit 502 is used to extract the multi-view visual features of each video clip frame in the video clip frame sequence according to a preset algorithm, and construct a clip-level graph for each video clip; the clip-level graph is used to represent the state of social relationships of people.

[0117] The determining unit 503 is used to process the clip-level image based on a first preset attention mechanism to determine scene-level character features and character pair features; wherein, the first preset attention mechanism is to include historical information evolution trends of each time period, and is used to make the scene-level character features and character pair features include time dimension information.

[0118] The capture unit 504 is used to determine the scene-level social relationship graph of the character based on the scene-level character features and character pair features, so as to capture the evolution of the character's social relationship in the target video.

[0119] In this embodiment, the response unit 501 generates a sequence of video clip frames with related relationships in response to the acquired target video. The clip-level graph generation unit 502 extracts multi-view features of each video clip frame in the video clip frame sequence according to a preset algorithm, generating a clip-level graph. The determination unit 503 processes the clip-level graph based on a first preset attention mechanism to determine scene-level character features and character pair features. The capture unit 504 uses the scene-level character features and character pair features to determine the scene-level social relationship graph of the characters. Thus, by processing the video clip frame sequence using the preset algorithm and the first preset attention mechanism, the acquired scene-level character features and character pair features contain multi-view and multi-modal information, and the features capture changes in the time dimension. This overcomes the technical problem of existing technologies that only consider character relationship recognition while ignoring temporal evolution factors, improving the accuracy of social relationship evolution capture.

[0120] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.

[0121] The device includes a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code to enable the device to perform the person relationship evolution capture method described in any embodiment of this application.

[0122] The computer storage medium stores code, and when the code is run, the device running the code implements the method for capturing the evolution of human relationships as described in any embodiment of this application.

[0123] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0124] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0125] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0126] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for capturing the evolution of human social relationships, characterized in that, The method includes: In response to the acquired target video, a sequence of video clip frames with related relationships is generated; the sequence of video clip frames includes characters. Based on a first preset algorithm, character features, character pair features, and segment video features are extracted from the video clip frame sequence; the segment video features include scene information. Based on the second preset algorithm, dialogue features are extracted from the dialogue files in the video clip frame sequence; wherein, the character features, character pair features, segment video features and dialogue features constitute multi-view visual features; The multi-view visual features are used as different nodes in the clip-level graph, and the information of the different nodes is propagated using a second preset attention mechanism. Different weights are assigned to the different nodes, and information is passed to adjacent nodes to generate a clip-level graph for each video clip; the clip-level graph is composed of the multi-view visual features; the second preset attention mechanism is a graph attention mechanism used to construct the clip-level graph; The clip-level image is processed based on a first preset attention mechanism to determine scene-level character features and character pair features; wherein, the first preset attention mechanism is to include historical information evolution trends of each time period, so that the scene-level character features and character pair features include time dimension information. Based on the scene-level character features and character pair features, a scene-level social relationship graph of the character roles is determined to capture the evolution of character social relationships in the target video.

2. The method according to claim 1, characterized in that, The step of determining the scene-level social relationship graph of the character based on the scene-level character features and character pair features includes: The scene-level character features and character pair features are input into a preset third preset attention mechanism to adjust the scene-level character pair features; The adjusted scene-level character pair features are sent to the fully connected layer to classify the social relationships of the character pairs and obtain the scene-level social relationship graph of the characters.

3. The method according to claim 2, characterized in that, The method further includes: Based on the determined scene-level social relationships and time-dimensional sliding window of the characters, scene-level social relationship graphs of the characters in different video clip frame sequences are generated. The scene-level social relationship graphs of the aforementioned characters are merged according to the chronological evolution order to generate a global evolutionary social relationship graph.

4. The method according to any one of claims 1-3, characterized in that, The step of generating a sequence of video clip frames with related relationships in response to the acquired target video includes: Acquire the target video; The target video is divided into multiple video clips based on the plot and scene; The multiple video clips are sampled at equal intervals to obtain a video clip frame sequence; The video clip frame sequence is subjected to character localization and re-identification to determine the correlation between video clip frames and generate a video clip frame sequence with correlation.

5. The method according to claim 4, characterized in that, The step of locating and re-identifying characters in the video clip frame sequence to determine the relationships between video clip frames includes: The video clip frames are detected based on a pre-trained person detector, and the person bounding boxes and their corresponding confidence scores are obtained. In response to the confidence level being greater than a first preset threshold, it is determined that the character frame contains a character. The face bounding boxes marked with identifiers are matched with the character bounding boxes containing the character roles, and the character bounding boxes are re-identified; Based on the third preset algorithm, the volume features of all character frames containing the character are extracted to determine the character frame features; The similarity between the person bounding box features and the labeled person bounding box features is calculated based on cosine similarity. If the similarity is greater than a preset threshold, the character frame and the labeled character frame are the same character.

6. A device for capturing the evolution of human social relationships, characterized in that, The device includes: A response unit is used to generate a sequence of video clip frames with related relationships in response to the acquired target video; the sequence of video clip frames includes characters. The clip-level image generation unit is used to extract character features, character pair features, and segment video features from the video clip frame sequence based on a first preset algorithm; the segment video features include scene information. Based on a second preset algorithm, dialogue features are extracted from the dialogue files in the video clip frame sequence; wherein, the character features, character pair features, segment video features, and dialogue features constitute multi-view visual features; the multi-view visual features are used as different nodes in the clip-level graph, and the information of the different nodes is propagated using a second preset attention mechanism; different weights are assigned to the different nodes, and information is passed to adjacent nodes to generate a clip-level graph for each video clip; the clip-level graph is composed of the multi-view visual features; the second preset attention mechanism is a graph attention mechanism used to construct the clip-level graph; The determining unit is used to process the clip-level image based on a first preset attention mechanism to determine scene-level character features and character pair features; wherein, the first preset attention mechanism is to include historical information evolution trends of each time period, and is used to make the scene-level character features and character pair features include time dimension information; The capture unit is used to determine the scene-level social relationship graph of the characters based on the scene-level character features and character pair features, so as to capture the evolution of the characters' social relationships in the target video.

7. A device for generating a method for capturing the evolution of human social relationships, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the steps of the method for capturing the evolution of human social relationships as described in any one of claims 1 to 5, according to the instructions in the program code.

8. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method for capturing the evolution of human social relationships as described in any one of claims 1 to 5.

Images