Role relation generation method and device, model training method and device and electronic equipment

Abstract

The invention provides a role relationship generation method and device, a model training method and device and electronic equipment. The method for determining the dynamic role relationship comprises the following steps: acquiring multi-modal summary information of each segmented video file of a video file; determining role feature information of the to-be-analyzed role according to the multi-modal summary information of all the segmented video files; processing the role feature information by using a role relationship generation model in a role relationship iterative input mode, and generating a segmented role relationship between the to-be-analyzed role and other roles, a support event corresponding to the segmented role relationship, and the number of segmented files where the support event is located; according to the segmented role relationship, the support event and the number of the segmented files, constructing a segmented role relationship graph of the to-be-analyzed role; and calling a role relationship prediction model to process the segmented role relationship atlas, and generating a role relationship sequence of the to-be-analyzed role and other roles in the video file, wherein the role relationship sequence comprises a plurality of continuous time sequences.

Description

Technical Field

[0001] This application relates to the field of video file processing technology, and in particular to a method for generating role relationships, a model training method, an apparatus, and an electronic device. Background Technology

[0002] Existing analyses of character relationships in TV dramas typically focus on static relationships (relatively stable connections between characters at a specific point in time or in a fixed scene, unchanging with the progression of the plot). Static relationships only reflect relationships at a fixed point in time and cannot capture dynamic changes as the plot unfolds. Some analyses also employ manual annotation of character relationships in TV dramas, but this method is relatively costly. Summary of the Invention

[0003] The purpose of this application is to provide a method for generating role relationships, a model training method, a device, and an electronic device, so as to automatically generate dynamic role relationships in video files without human intervention. The specific technical solution is as follows: In a first aspect of this application, a method for generating role relationships is provided, comprising: Obtain multimodal summary information for each segment of the video file, wherein the multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file; Based on the multimodal summary information of all segmented video files of the video file, determine the character feature information of the character to be analyzed in the video file; Using an iterative input approach based on role relationships, a role relationship generation model is used to process the role feature information, generating segmented role relationships between the role to be analyzed and other roles in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located. The segmented role relationships represent the relationship between the role to be analyzed and other roles in the segmented video files, and the supporting events are events extracted from the segmented video files to characterize the relationship between the role to be analyzed and other roles. Based on the segmented role relationships, the supporting events, and the number of segmented files, a segmented role relationship graph of the role to be analyzed is constructed. The segmented character relationship graph is processed by calling a pre-trained character relationship prediction model to generate a sequence of multiple temporally continuous character relationships between the character to be analyzed and other characters in the video file.

[0004] A second aspect of this application provides a model training method, comprising: The multimodal summary information of each segment of the sample video file is obtained. The multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. Based on the multimodal summary information of all segmented video files of the sample video file, the character feature information of the target character in the sample video file is determined; wherein, the character feature information includes: character introduction information and character experience information, wherein, the character introduction information is the static attribute information of the target character extracted from the multimodal summary information of the sample video file, and the character experience information is the temporally continuous character event sequence of the target character extracted from the multimodal summary information of the sample video file; The character profile information, character experience information, and historical segmented character relationships of the specified number of segments are input into the character relationship generation model. The model then obtains the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships for the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships for the video file. Based on the sample segmented role relationships, the target supporting events, and the number of segmented files in which the target supporting events are located, a sample relationship graph of the target role is constructed, and the real role relationship sequence corresponding to the sample relationship graph is labeled. The sample relationship graph is input into the role relationship prediction model to be trained, and the predicted role relationship sequence corresponding to the sample relationship graph output by the relationship prediction model to be trained is obtained. The loss value of the role relationship prediction model to be trained is calculated based on the real role relationship sequence and the predicted role relationship sequence. If the loss value indicates that the role relationship prediction model to be trained has converged, the trained role relationship prediction model is obtained.

[0005] In a third aspect of this application, a role relationship generation apparatus is provided, comprising: The summary acquisition module is used to acquire multimodal summary information of each segment of the video file. The multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. The feature determination module is used to determine the role feature information of the character to be analyzed in the video file based on the multimodal summary information of all segmented video files of the video file; The relationship generation module is used to process the character feature information using a role relationship generation model through iterative input of role relationships, and to generate segmented role relationships between the character to be analyzed and other characters in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located; the segmented role relationships represent the relationship between the character to be analyzed and other characters in the segmented video files, and the supporting events are events extracted from the segmented video files to characterize the relationship between the character to be analyzed and other characters; The graph construction module is used to construct a segmented role relationship graph of the role to be analyzed based on the segmented role relationships, the supporting events, and the number of segmented files. The relationship sequence generation module is used to call a pre-trained role relationship prediction model to process the segmented role relationship graph and generate multiple temporally continuous role relationship sequences between the role to be analyzed and other roles in the video file.

[0006] In a fourth aspect of this application, a model training apparatus is provided, comprising: The sample summary acquisition module is used to acquire sample multimodal summary information for each segment of the sample video file. The sample multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. The role feature determination module is used to determine the role feature information of the target role in the sample video file based on the sample multimodal summary information of all segmented video files of the sample video file; wherein, the role feature information includes: role introduction information and role experience information, wherein the role introduction information is static attribute information of the target role extracted from the sample multimodal summary information, and the role experience information is a temporally continuous sequence of role events of the target role extracted from the sample multimodal summary information; The segmented relationship acquisition module is used to input character introduction information, character experience information, and historical segmented character relationships into the character relationship generation model, which contains a set number of segments. It then acquires the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located, all output by the character relationship generation model. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships for the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships for the video file. The sample graph construction module is used to construct a sample relationship graph of the target role based on the sample segment role relationship, the target supporting event, and the number of segment files in which the target supporting event is located, and to label the real role relationship sequence corresponding to the sample relationship graph; The sample relationship graph input module is used to input the sample relationship graph into the role relationship prediction model to be trained, and to obtain the predicted role relationship sequence corresponding to the sample relationship graph output by the relationship prediction model to be trained; The loss value calculation module is used to calculate the loss value of the role relationship prediction model to be trained based on the real role relationship sequence and the predicted role relationship sequence. The prediction model acquisition module is used to obtain the trained role relationship prediction model when the loss value indicates that the role relationship prediction model to be trained has converged.

[0007] In another aspect of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the aforementioned role relationship generation methods or model training methods.

[0008] In another aspect of this application, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform any of the role relationship generation methods or model training methods described above.

[0009] In another aspect of this application, a computer program product containing instructions is also provided, on which a computer program is stored, which, when run on a computer, causes the computer to execute any of the role relationship generation methods or model training methods described above.

[0010] The solution provided in this application generates segmented character relationships of the characters to be analyzed by acquiring multimodal summary information of each episode of a video file, and constructs a fine-grained segmented character relationship graph. Based on a pre-trained character relationship prediction model, the fine-grained segmented character relationship graph is processed to generate multiple temporally continuous character relationship sequences. This process requires no manual intervention and achieves the goal of automatically generating dynamic character relationships. Furthermore, compared to manual annotation, it reduces labor costs. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0012] Figure 1 A flowchart illustrating the steps of a role relationship generation method provided in this application embodiment; Figure 2 A flowchart illustrating the steps of a multimodal summary information generation method provided in this application embodiment; Figure 3 A flowchart illustrating the steps of a method for obtaining segmented video files provided in this application embodiment; Figure 4 A flowchart illustrating the steps of another method for obtaining segmented video files provided in this application embodiment; Figure 5 A flowchart illustrating the steps of another multimodal summary information generation method provided in this application embodiment; Figure 6 A flowchart illustrating the steps of a method for determining scene inflection points provided in this application embodiment; Figure 7 A flowchart illustrating the steps of a segmented role relationship generation method provided in this application embodiment; Figure 8 A flowchart illustrating the steps of a map construction method provided in this application embodiment; Figure 9 A flowchart illustrating the steps of a recommended video filtering method provided in this application embodiment; Figure 10 A flowchart illustrating the steps of a model training method provided in this application embodiment; Figure 11 This is a schematic diagram of the structure of a role relationship generation device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0014] Figure 1 A flowchart illustrating the steps of a role relationship generation method provided in this application embodiment is shown below. Figure 1 As shown, the method for generating role relationships may include steps 101 to 105.

[0015] Step 101: Obtain multimodal summary information for each segment of the video file. The multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file.

[0016] In this embodiment, multimodal summary information refers to a comprehensive summary formed by integrating the character information (such as character identity, number of appearances, interactive objects, character coordinates, etc.) within a single segment of a video file with the dialogue (including dialogue text and tone-related text conversion information) of that segment. It integrates multi-dimensional information such as visual (characters) and text (dialogue), and is a condensation of the core content of a single segment of a video file.

[0017] When generating dynamic character relationships in video files (such as TV drama text, short video files, etc.), multimodal summary information of each segment of the video file can be obtained. This multimodal summary information can be obtained by integrating character information (character coordinates, facial recognition results, etc.) within each segment of the video file, as well as the dialogue within each segment of the video file (i.e., character dialogue within the video images). Specifically, multiple segmented video files can be obtained, and the final multimodal summary information can be generated based on the extracted character information and dialogue information of each segment. This implementation process will be described in detail in the following embodiments, and will not be repeated here.

[0018] After obtaining the multimodal summary information of each segment of the video file, proceed to step 102.

[0019] Step 102: Based on the multimodal summary information of all segmented video files of the video file, determine the character feature information of the character to be analyzed in the video file.

[0020] Character feature information refers to the characteristic information used to describe the character to be analyzed in the video file. In this example, character feature information may include: character introduction information and character experience information.

[0021] The character experience information can be static attribute information of the character to be analyzed, extracted from the multimodal summary information. Specifically, it can be a comprehensive description of the basic attributes and core characteristics of the character to be analyzed (such as character A and character B) extracted from the multimodal plot synopsis of all segmented video files. It focuses on the relatively stable settings of the character, is a summary of the character's "essence", does not change frequently with the dynamic changes of the plot, and provides a basic reference for understanding the character's subsequent behavior and relationships.

[0022] Character experience information can be a temporally continuous sequence of character events extracted from multimodal summary information. Specifically, based on the multimodal summary information of all segmented video files, the key development process of the character to be analyzed in the entire series can be sorted out, arranged chronologically (episode order), including important events (such as "framed in episode 5" or "allyed with a certain character in episode 20"), status changes (such as "promoted from a commoner to a general" or "from trust to betrayal"), and other dynamic trajectories.

[0023] After obtaining the multimodal summary information for each segment of the video file, the character feature information of the character to be analyzed in the video file can be determined based on the multimodal summary information of all video segments. Specifically, the attributes of the character to be analyzed can be extracted from the multimodal summary information to obtain the character's profile information. At the same time, the character's experience information can be sorted out according to the multimodal summary information, that is, the key events can be integrated according to the chronological order of the video segments to obtain the character's experience information.

[0024] After obtaining the character feature information of the character to be analyzed, proceed to step 103.

[0025] Step 103: Using an iterative input method based on role relationships, the role feature information is processed using a role relationship generation model to generate segmented role relationships between the role to be analyzed and other roles in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located; the segmented role relationships represent the relationship between the role to be analyzed and other roles in the segmented video files, and the supporting events are events extracted from the segmented video files to characterize the relationship between the role to be analyzed and other roles.

[0026] The Role Relationship Generation Model is an intelligent model that automatically analyzes and generates relationships between roles based on multimodal information and dynamic trajectories of roles. Its core function is to receive role characteristic information, such as role profile information (static attributes), role experience information (dynamic event trajectories), and multimodal summaries (scenes, dialogue, role interactions) from individual segmented video files. By understanding the role's behavioral logic, causal relationships between events, and the evolution of relationships, it outputs segmented role relationships between roles, the supporting events corresponding to these segmented relationships, and the number of segmented files containing each supporting event.

[0027] Among them, segmented role relationships can be used to represent the relationship between the role to be analyzed and other roles in segmented video files.

[0028] Supporting events can be events extracted from segmented video files that characterize the relationship between the character being analyzed and other characters.

[0029] Iterative input of role relationships refers to the process of progressively inputting role introductions and experiences in segments of video files when analyzing role relationships. By combining the role relationship results of the processed segmented video files, the relationship judgment of subsequent segmented video files is dynamically updated (i.e., the relationship of the first few segmented video files will affect the relationship analysis of the later segmented video files), thus realizing the progressive derivation of relationships.

[0030] Segmented character relationships refer to the specific connections (such as "hostility", "cooperation", "unrequited love") between the character to be analyzed and other characters in certain segments, which may change as the plot progresses.

[0031] Supporting events refer to specific plot events used to prove the existence of a relationship between characters in a certain segment. They are "evidence" of the relationship (such as "A saved B in the 8th segment video file, supporting the 'ally' relationship between A and B").

[0032] After obtaining the character feature information of the character to be analyzed, a character relationship iterative input method can be used to process the character feature information using a character relationship generation model. This generates segmented character relationships between the character to be analyzed and other characters in the video file, supporting events corresponding to the segmented character relationships, and the number of segmented files in which the supporting events are located. The implementation process will be described in detail in the following embodiments, and will not be repeated here.

[0033] After obtaining the segmented role relationships, supporting events, and the number of segmented files, proceed to step 104.

[0034] Step 104: Based on the segmented role relationships, the supporting events, and the number of segmented files, construct the segmented role relationship graph of the role to be analyzed.

[0035] A segmented role relationship graph is a graph constructed according to the order of segmented video files (such as video files 1-5, video files 6-10, etc.). Each few video files correspond to a subgraph, which contains the segmented role relationship between the role to be analyzed and other roles, as well as the supporting events corresponding to the segmented role relationship.

[0036] After obtaining the segmented character relationships, supporting events, and the number of segmented files, a segmented character relationship graph of the character to be analyzed can be constructed based on these factors. Specifically, the segmented character relationships, supporting events, and the number of segmented files obtained above for a set number of segmented video files can be visualized in chronological order to generate a segmented character relationship graph. For example, taking a single episode of a TV series as an example, the subgraph for episodes 1-5 shows nodes (character A and character B), edges ("friends"), and edge labels indicating supporting events: mutual support. The subgraph for episodes 5-10 shows nodes (character A and character B), edges ("enemies"), and edge labels indicating supporting events: conflict. The overall graph is arranged chronologically according to the episodes, clearly showing the relationship network for each few episodes.

[0037] It is understood that the above examples are merely examples listed to better understand the technical solutions of the above embodiments, and are not intended to be the only limitation of this embodiment.

[0038] The detailed process of map construction will be described in detail in the following embodiments, and will not be repeated here.

[0039] After constructing the segmented role relationship graph of the role to be analyzed, proceed to step 105.

[0040] Step 105: Call the pre-trained role relationship prediction model to process the segmented role relationship graph and generate a sequence of multiple temporally continuous role relationships between the role to be analyzed and other roles in the video file.

[0041] The role relationship prediction model refers to a pre-trained model used to generate multiple temporally consecutive role relationship sequences in a video file. The training process of this model will be described in detail in the following embodiments, and will not be repeated here.

[0042] A sequence of multiple time-series continuous character relationships (i.e., dynamic character relationships) refers to character relationships that evolve as the plot progresses (e.g., across series) based on a segmented character relationship graph and predicted by a model. It includes the nodes of relationship change, the reasons for change, and the trend (e.g., "from hostility → temporary cooperation → complete hostility").

[0043] After constructing the segmented role relationship graph of the role to be analyzed, a pre-trained role relationship prediction model can be called to process the segmented role relationship graph, generating multiple temporally continuous role relationship sequences between the role to be analyzed and other roles in the video file. Specifically, the constructed segmented role relationship graph can be input into the pre-trained role relationship prediction model. The model analyzes the relationship change trends across segmented video files and outputs multiple temporally continuous role relationship sequences. For example, the dynamic role relationship between role 1 and role 2 may be: from high school classmates to accomplices, then to enemies, and finally to a complex relationship of mutual exploitation and dependence.

[0044] This application embodiment generates segmented role relationships of the characters to be analyzed by acquiring multimodal summary information from each segment of the video file, and constructs a fine-grained segmented role relationship graph. Based on a pre-trained role relationship prediction model, the fine-grained segmented role relationship graph is processed to generate multiple temporally continuous role relationship sequences (i.e., dynamic role relationships). This process requires no manual intervention, achieving the goal of automatically generating dynamic role relationships. Furthermore, compared to manual annotation, it reduces labor costs.

[0045] Next, combined Figure 2 The process of generating multimodal summary information is described in detail.

[0046] Reference Figure 2 The diagram illustrates a flowchart of a multimodal summary information generation method provided in an embodiment of this application. Figure 2 As shown, the multimodal summary information generation method may include steps 201 to 203.

[0047] Step 201: Segment the video file to obtain segmented video files.

[0048] In this embodiment, when generating the dynamic character sequence of the video file, the video file can be segmented to obtain segmented video files.

[0049] In specific implementations, video files can be segmented according to scene transition points or video episodes, etc. The specific segmentation methods will be described in detail in the following embodiments, and will not be repeated here.

[0050] Step 202: Extract video frames from each segmented video file according to the first time interval to obtain the basic video frames of each segmented video file.

[0051] In this embodiment, the first time interval refers to the fixed time interval (such as 1 second, 2 seconds, etc.) used when extracting basic video frames from each segment of the video file.

[0052] The basic video frame refers to the set of video frames extracted from each segmented video file according to the first time interval, which is the basic material for subsequent analysis of scene changes.

[0053] When obtaining multimodal summary information for each segment of a video file, video frames can be extracted from each segment according to a first time interval to obtain the base video frame for each segment. For example, if each segment of a video file is 5 minutes long and the first time interval is 1 second, then 5 × 60 video frames need to be extracted from each segment, that is, one video frame is extracted every 1 second, to serve as the base video frame for each segment.

[0054] After extracting the basic video frames of each segment of the video file according to the first time interval, step 203 is executed.

[0055] Step 203: Generate multimodal summary information for each segmented video file based on the character information and video dialogue information in each of the basic video frames.

[0056] After obtaining the basic video frames of each segmented video file, multimodal summary information for each segmented video file can be generated based on the character information and dialogue information in each basic video frame. Specifically, scene transition points of each segmented video file can be obtained, resulting in multiple video scenes for each segmented video file. A summary of each video scene can be generated based on the character information and dialogue information of these multiple video scenes. This summary is then processed using a large language model to obtain the multimodal summary information for each segmented video file. This implementation process will be described in detail in the following embodiments, and will not be repeated here.

[0057] This application embodiment achieves efficient structured processing of video content by segmenting the video, extracting basic frames at fixed intervals, and then combining the in-frame character information and video dialogue to generate segmented multimodal summaries. This allows for the accurate extraction of core information from each segment, making key video content easier to quickly obtain and utilize, and improving the efficiency and accuracy of video information processing.

[0058] Next, combined Figure 3 The process of segmenting video files based on scene transition points is described in detail.

[0059] Reference Figure 3The diagram illustrates a flowchart of a method for obtaining segmented video files according to an embodiment of this application. Figure 3 As shown, the method for obtaining segmented video files may include steps 301 and 302.

[0060] Step 301: Determine the scene inflection point of the video file based on the similarity between two adjacent keyframes in the video file.

[0061] In this embodiment, a video keyframe refers to a frame that can represent the core content of a certain segment of the video. It does not need to be selected frame by frame. It is usually a frame with representative content features (such as screen elements and color distribution) that can reflect the main information of the video within that time period.

[0062] Similarity is a quantitative indicator that measures the degree of similarity between two adjacent keyframes in terms of image content (color, objects, scene layout), calculated by a specific algorithm (such as visual feature comparison). The value range is usually 0-1 (1 means completely identical, 0 means completely different).

[0063] A scene inflection point refers to a frame position where the similarity between adjacent keyframes is lower than a preset threshold. The video content before and after this position changes significantly (such as scene switching, camera jumps, or subject changes), and it is the dividing point for different scenes.

[0064] When segmenting a video file, the similarity between two adjacent keyframes can be obtained. Specifically, visual feature algorithms can be selected to extract and compare features from adjacent keyframes to calculate their similarity. Examples include color histogram comparison and feature point matching.

[0065] After calculating the similarity, the scene inflection point in the video file can be determined based on the similarity between two adjacent keyframes. Specifically, a similarity threshold can be set (e.g., 0.5, which can be adjusted according to the video type: lower thresholds for movies and TV shows, and higher thresholds for surveillance videos). The similarity values ​​of all adjacent keyframes are iterated. When the similarity of a certain group of adjacent keyframes is lower than the threshold, the position of the next keyframe is determined as the scene inflection point. At the same time, "instantaneous interference" (such as low similarity in a single frame caused by screen flicker) is excluded. Only when the similarity of 2-3 consecutive groups of adjacent frames is lower than the threshold is the scene inflection point finally confirmed.

[0066] Step 302: Segment the video file according to the scene turning point to obtain the segmented video file.

[0067] After determining the scene transition points, the video file can be segmented based on these points to obtain segmented video files. For example, a segment boundary can be defined as the distance from one transition point (or the start of the video) to the next: First segment: from the video start time (00:00:00) to the timestamp of the first scene transition point. Middle segment: between the timestamps of two adjacent scene transition points. Last segment: from the timestamp of the last scene transition point to the end of the video, etc.

[0068] It is understood that the above examples are merely examples listed to better understand the technical solutions of the embodiments of this application, and are not intended to be the only limitation on the embodiments.

[0069] This application embodiment uses keyframe similarity to determine scene changes and achieves video segmentation. Compared with segmentation at fixed time intervals, it is more in line with the logic of actual scenes and can accurately capture the natural boundaries of the screen content. The generated segmented video files all correspond to the complete scene, effectively improving the rationality and accuracy of video segmentation and laying a high-quality data foundation for subsequent scene-level content analysis (such as character relationship extraction and scene summary generation).

[0070] Next, combined Figure 4 The process of segmenting video files is described in detail.

[0071] Reference Figure 4 This illustrates a flowchart of another method for obtaining segmented video files provided in an embodiment of this application. Figure 4 As shown, the method for obtaining segmented video files may include steps 401 and 402.

[0072] Step 401: Obtain the video episodes of the video file.

[0073] In this embodiment, a video series refers to an independent unit that is naturally present in a video file and is logically divided according to its content. It is usually a content segmentation format pre-set by the creator. Common types include single episodes of TV series / web series, single episodes of short video series, and single chapters of documentaries. Each series has a relatively complete plot or theme and usually has a clear episode number identifier (such as episode 1, episode 5, etc.).

[0074] When segmenting a video file, you can obtain the video episodes from the video file.

[0075] Step 402: Segment the video file according to the video episode to obtain segmented video files, each segmented video file corresponding to one episode of the video file.

[0076] After obtaining the video episodes from the video file, the video file can be segmented according to the video episodes to obtain segmented video files, with each segmented video file corresponding to one episode of the video file.

[0077] This application embodiment uses video series as the original content unit for segmentation, eliminating the need for complex image feature calculations. It can quickly achieve a precise correspondence between video and episode content, and the generated segmented files naturally conform to users' understanding of episodes and viewing habits. At the same time, it provides clear and regular content units for subsequent episode-level processing (such as segmented character relationship analysis and episode summary generation), greatly reducing the complexity of subsequent processing.

[0078] Next, combined Figure 5 The process of generating multimodal summary information for segmented video files is described in detail.

[0079] Reference Figure 5 The diagram illustrates a flowchart of another multimodal summary information generation method provided in an embodiment of this application. Figure 5 As shown, the multimodal summary information generation method may include steps 501 to 504.

[0080] Step 501: Determine the scene inflection point of each segmented video file based on the similarity between two adjacent basic video frames in each segmented video file.

[0081] In this embodiment, the scene transition point refers to the key time point in the video where the scene changes (such as switching from "indoor study" to "outdoor courtyard"), which is determined by the similarity difference between adjacent basic video frames.

[0082] After obtaining the basic video frames, the similarity between two adjacent basic video frames of each segmented video file can be acquired, and the scene transition points of each segmented video file can be determined based on the similarity between two adjacent basic video frames. This implementation process will be described in detail in the following embodiments, and will not be repeated here.

[0083] Step 502: According to the second time interval, extract video frames from the basic video frames between two adjacent scene transition points of each segmented video file to obtain the target video frames of all video scenes of each segmented video file; the video frames between two adjacent scene transition points constitute a video scene.

[0084] The second time interval refers to the time interval used when extracting target video frames within each video scene (between two adjacent scene transition points), and is used to capture key visual details within the scene. In this example, the first time interval and the second time interval can be the same time interval or different time intervals; this embodiment does not impose any restrictions on this.

[0085] The target video frame refers to the video frame extracted from each video scene according to the second time interval, and is the key frame representing the core content of the scene.

[0086] A video scene refers to a continuous video segment between two adjacent scene transition points, belonging to the same spatial environment or plot unit.

[0087] After determining the scene transition points of each segmented video file, video frames can be extracted from the basic video frames between two adjacent scene transition points of each segmented video file according to the second time interval, resulting in the target video frames for all video scenes of each segmented video file. The video frames between two adjacent scene transition points constitute a video scene, and no two adjacent video scenes are identical.

[0088] Step 503: Generate video summary information corresponding to each video scene based on the character information and video dialogue information in the target video frame of each video scene.

[0089] Character information refers to character-related information identified from target video frames, including character identity, number, and interaction status (such as "looking at each other" or "standing") (usually obtained through facial recognition and posture recognition technologies).

[0090] Video dialogue information refers to the text content of the dialogue between characters in a video scene (which can be obtained by converting audio dialogue into text using speech-to-text technology and associating it with the corresponding characters and time points. Of course, it can also be obtained from the text subtitles on the video screen).

[0091] Video summary information refers to a scene-level summary formed by integrating character information and dialogue information from the target video frame for a single video scene (such as "Scene: Imperial Garden; Characters: A, B; Dialogue: A: 'The weather is very nice today,' B: 'It is indeed a good time to go out', etc.").

[0092] After obtaining the target video frames, video summary information for each video scene can be generated based on the character information and dialogue information in the target video frames of each video scene. Specifically, the transition point Si of the shot can be obtained, one frame per second can be extracted from the video from Si to Si+1, and the coordinates of the characters in each frame can be detected by face recognition. The dialogue of this video can also be obtained, and this data can be input into a multimodal large model to generate a video description for the shot. A video summary is generated for each shot.

[0093] Step 504: Using a large language model, process the splicing summary information of each segmented video file to obtain the multimodal summary information of each segmented video file. The splicing summary information is the summary information obtained by splicing the video summary information of all video scenes of each segmented video file.

[0094] The concatenated summary information can be obtained by concatenating the video summary information of all video scenes in each segmented video file.

[0095] After generating video summary information for each video scene, a large language model can be used to process the spliced ​​summary information of each segmented video file to obtain multimodal summary information for each segmented video file.

[0096] This application embodiment first locates the scene turning points of segmented videos, then extracts target frames within the scene to generate scene summaries, and finally integrates and splices the summaries using a large language model. This enables layer-by-layer refinement of "segmentation-scene-multimodal summarization," which not only accurately captures the core scene-level information of the video, but also improves the logic and completeness of the summaries through a large model, making the multimodal summaries of segmented videos more consistent with the content logic and more comprehensive in information.

[0097] Next, combined Figure 6 The process for determining the turning point of a scene is described in detail.

[0098] Reference Figure 6 The diagram illustrates a flowchart of a method for determining scene inflection points according to an embodiment of this application. Figure 6 As shown, the method for determining the turning point of this scenario may include steps 601, 602 and 603.

[0099] Step 601: Obtain the similarity between two adjacent base video frames of each segmented video file.

[0100] In this embodiment, after obtaining the basic video frames of each segmented video file, the similarity between two adjacent basic video frames of each segmented video file can be obtained. That is, for the basic video frames that have been extracted at a first time interval (e.g., every 1 second), they are grouped in pairs according to time order (frame 1 and frame 2, frame 2 and frame 3, ... frame (n-1) and frame n), and the similarity of each group of adjacent frames is calculated using computer vision algorithms (e.g., Structural Similarity Index SSIM, hash algorithm).

[0101] Step 602: Obtain two adjacent target base video frames with a similarity less than the similarity threshold.

[0102] After obtaining the similarity between two adjacent base video frames, two adjacent target base video frames with a similarity less than the similarity threshold can be obtained.

[0103] Step 603: Determine the preceding target base video frame of the two adjacent target base video frames as the scene inflection point.

[0104] Then, the preceding target video frame between two adjacent target base video frames can be determined as the scene inflection point.

[0105] In practical applications, for each segment of a TV series video file, one frame can be extracted per second, and the similarity between frames Fi and Fi+1 can be calculated. When the similarity is less than a threshold (e.g., 0.9), it indicates that a change in the shot has occurred. The time of the Fi pair is then defined as the transition point of the shot, and the transition points Si,...Sj of the entire episode are calculated.

[0106] This application embodiment quantifies the similarity differences between adjacent video frames, which can automatically and accurately identify scene turning points in video series, realize scene-level segmentation of video content, lay the foundation for subsequent extraction of target video frames by scene and generation of scene summaries, ensure that the content analysis of each scene focuses on the same spatiotemporal unit, avoid cross-scene information confusion, and ultimately improve the accuracy and structure of multimodal summary information.

[0107] Next, combined Figure 7 The process of obtaining segmented role relationships is described in detail.

[0108] Reference Figure 7 The diagram illustrates a flowchart of a segmented role relationship generation method provided in an embodiment of this application. Figure 7 As shown, the segmented role relationship generation method may include steps 701 and 702.

[0109] Step 701: During the process of using the character relationship generation model to predict relationships according to the segmented video files from front to back, each time the model inference is performed, the character introduction information of the set number of segments, the character experience information, and the historical segmented character relationships are input into the character relationship generation model; wherein, the historical segmented character relationships refer to the segmented character relationships output by the previous model inference during the character relationship generation model's prediction of character relationships for the video files, and the historical segmented character relationships when the character relationship generation model first performs character relationship prediction for the video files are empty.

[0110] In this embodiment, when reasoning about the segmented role relationships of each segmented video file, during the relationship prediction process using the role relationship generation model in the order of the segmented video files from beginning to end, the character introduction information, character experience information, and historical segmented role relationships of a set number of segments are input into the role relationship generation model each time the model is inferred. The historical segmented role relationships refer to the segmented role relationships output by the model in the previous model inference during the role relationship prediction process of the video file. The historical segmented role relationships are empty when the role relationship generation model first performs role relationship prediction on the video file.

[0111] Step 702: Obtain the segmented role relationship by processing the character profile information, the character experience information, and the segmented role relationship output by the previous generation model, and obtain the segmented role relationship, the supporting events corresponding to the segmented role relationship, and the number of segmented files in which the supporting events are located.

[0112] Simultaneously, the model can process character introduction information, character experience information, and the segmented character relationships previously output by the character relationship generation model to obtain segmented character relationships, corresponding supporting events, and the number of segmented files containing the supporting events. For example, for segmented video files, taking each video episode as an example, five episodes can be input at a time. The first input is episodes 1-5, the second input is episodes 6-10, and so on. During the first model inference, the input is the character introduction information and character experience information from episodes 1-5, resulting in the segmented character relationships for episodes 1-5. The second input is the character introduction information and character experience information from episodes 6-10, using the historical segmented character relationships from episodes 1-5 to obtain the segmented character relationships for episodes 6-10. This process continues until all episodes of the entire video file have been processed.

[0113] It is understood that the above examples are merely examples listed to better understand the technical solutions of the embodiments of this application, and are not intended to be the only limitation on the embodiments.

[0114] This application embodiment dynamically adjusts the current prediction based on historical relationships to ensure that the evolution of character relationships conforms to the plot logic, avoids isolated judgments, and ultimately generates a coherent and accurate segmented character relationship chain.

[0115] Next, combined Figure 8 The process of constructing the segmented role relationship graph is described in detail.

[0116] Reference Figure 8 The diagram illustrates a flowchart of the steps involved in a map construction method provided in an embodiment of this application. Figure 8 As shown, the map construction method may include steps 801 to 804.

[0117] Step 801: Determine the temporal order of the segment roles based on the segment sequence of the video file.

[0118] In this embodiment, after obtaining the segmented role relationships, the temporal order of the segmented role relationships can be determined according to the segmented order of the video file.

[0119] Step 802: Based on the temporal order of the segmented role relationships, determine whether there is a relationship conflict between each segmented role relationship and the segmented role relationships of the N adjacent segmented video files; wherein, the relationship conflict refers to the situation where the change magnitude of the current segmented role relationship and the N adjacent segmented role relationships is greater than a threshold, and there are no supporting events representing the change of segmented role relationships among the supporting events of the current segmented role relationship and the N adjacent segmented role relationships, where N is a positive integer.

[0120] Then, according to the temporal order of the segmented role relationships, it can be determined whether there are any contradictions between each segmented role relationship and the segmented role relationships of the N adjacent segmented video files. A contradiction occurs when the change in the current segmented role relationship compared to the N adjacent segments (previous and / or subsequent) exceeds a threshold, and there are no supporting events representing changes in the segmented role relationships among the supporting events for the current segmented role relationship and the N adjacent segments (previous and / or subsequent), where N is a positive integer. For example, if episode 1 represents brothers, episode 2 represents strangers, and episode 3 represents brothers, then the relationship in episode 2 contradicts the relationships before and after it and needs correction.

[0121] Step 803: If there is a relationship contradiction, iteratively execute the steps of using the role relationship iterative input method, using the role relationship generation model to process the role feature information, and generating the segmented role relationship between the role to be analyzed and other roles in the video file, the supporting events corresponding to the segmented role relationship, and the number of segmented files in which the supporting events are located, until there is no relationship contradiction, and obtain the corrected segmented role relationship.

[0122] If there are contradictions, step 103 can be iterated until there are no contradictions, thus obtaining the corrected segmented role relationships.

[0123] Step 804: Based on the corrected segmented role relationships, the supporting events, and the number of segmented files, construct the segmented role relationship map of the role to be analyzed.

[0124] After obtaining the corrected segmented role relationships, a segmented role relationship graph of the role to be analyzed can be constructed based on the corrected segmented role relationships, supporting events, and number of segments.

[0125] This application's embodiments correct inconsistencies in segmented character relationships, avoiding distortions caused by model misjudgments or information omissions, thus ensuring that the character relationships in each segment better align with the actual plot. Secondly, it guarantees the continuity of character relationship evolution, organizing and correcting character relationships according to segmented order, making changes in character relationships traceable and consistent with the audience's cognitive logic of plot development. Finally, it enhances the reliability of the segmented character relationship map. This map integrates corrected character relationships, supporting events, and the number of segmented files, providing high-quality foundational data for subsequent in-depth analysis of dynamic changes in character relationships and prediction of character relationship development trends, reducing analytical and predictive biases caused by data inconsistencies.

[0126] Next, combined Figure 9 The process for selecting recommended videos is described in detail.

[0127] Reference Figure 9 The diagram illustrates a flowchart of a recommended video filtering method provided in an embodiment of this application. Figure 9 As shown, the recommended video filtering method may include steps 901 to 906.

[0128] Step 901: Call the role relationship quantification model and use a mapping table to process the segmented role relationship graph to obtain the first value of each role relationship in the role relationship sequence in the preset quantification dimension, and weight the first value of the preset quantification dimension to obtain the first quantification index value sequence corresponding to the role relationship sequence. The mapping table records the value corresponding to the preset emotional quantification dimension, which includes at least one of the following: emotional dimension, trust dimension and interaction dimension.

[0129] In this embodiment, the role relationship quantification model is a model that can convert role relationships into quantifiable numerical indicators, and through this model, comparisons of different role relationships can be achieved.

[0130] The first quantified numerical sequence refers to an ordered set of values ​​formed by weighting the multi-dimensional values ​​of the corresponding role relationships and arranging them in chronological order.

[0131] After obtaining the segmented role relationship graph of the video file, the role relationship quantification model can be called to process the segmented role relationship graph using a mapping table. This yields the first value of each role relationship in the role relationship sequence in a preset quantification dimension. The first values ​​of the preset quantification dimensions are then weighted to obtain the first quantification index value sequence corresponding to the role relationship sequence. The specific implementation process can be as follows: 1. Determine the segmented role relationship graph to be analyzed and clarify the type of each role relationship in the graph (e.g., Role A - Role B: Cooperation). 2. Mapping table matching: Call the role relationship quantification model to match each role relationship in the graph with the mapping table, obtaining its first value in the preset quantification dimension. For example, a cooperative relationship receives +4 in the emotion dimension, +3 in the trust dimension, and +5 in the interaction dimension. 3. Multi-dimensional weighted values: Based on the importance of each quantification dimension (e.g., emotion weight 0.4, trust weight 0.3, interaction weight 0.3), calculate the comprehensive weighted value of each role relationship (e.g., 4×0.4 + 3×0.3 + 5×0.3 = 4.0). 4. Generate numerical sequence: Arrange the comprehensive weighted values ​​of all role relationships in chronological order in the role relationship sequence to obtain the numerical sequence of the first quantitative index (e.g., [3.2, 4.0, 3.8]).

[0132] Step 902: Obtain the basic role relationship graph of the target character in the video file to be recommended. The basic role relationship graph is constructed based on the segmented role relationships, supporting events, and number of segmented files of the target character in the video file to be recommended.

[0133] The basic role relationship graph refers to the initial relationship graph constructed for the target role in the recommended video. It is composed of the segmented role relationships of the target role in the video, supporting events (such as "the mutual assistance event in episode 5"), and the number of segmented files (such as 10 segments), reflecting the overall relationship between the target role and other roles.

[0134] When recommending video files, a basic role relationship graph of the target characters in the video files to be recommended can be obtained. This basic role relationship graph can be constructed based on the segmented role relationships, supporting events, and number of segmented files of the target characters in the video files to be recommended. The specific construction process can be referred to the description of steps 101 to 104 above, and will not be repeated here in this embodiment.

[0135] Step 903: Call the role relationship prediction model to extract the temporal association information between the target role and other roles in the video file to be recommended from the basic role relationship graph, and use a preset relationship type library to perform temporal prediction on the temporal association information to obtain multiple temporally continuous basic role relationship sequences between the target role and other roles in the video file to be recommended; wherein, the basic role relationship sequence includes: the role relationship type corresponding to each temporal segment.

[0136] Temporal correlation information refers to the logical changes in the relationship between the target role and other roles in different temporal segments (such as segment 1 → segment 2 → segment 3) in the basic role relationship graph (such as "from stranger to cooperation" and "from trust to hostility").

[0137] The preset relationship type library refers to a collection containing all possible role relationship types (such as "stranger, neutral, cooperative, trusting, hostile"), used to standardize the relationship output type for time series prediction.

[0138] The basic role relationship sequence refers to the time-series continuous set of role relationships output by the role relationship prediction model. Each sequence corresponds to the relationship type between the target role and other roles in different time segments, such as "Role A - Role C: Strangers (segment 1) → Neutral (segment 2) → Cooperative (segment 3)".

[0139] After obtaining the basic role relationship graph, a role relationship prediction model can be used to extract the temporal association information between the target role and other roles in the video file to be recommended. A preset relationship type library is then used to perform temporal prediction on this temporal association information, resulting in multiple temporally continuous basic role relationship sequences between the target role and other roles in the video file to be recommended. Each basic role relationship sequence includes the role relationship type corresponding to each temporal segment.

[0140] In practical implementation, a role relationship prediction model can be used to analyze the changes in the relationship between the target role and other roles in the basic role relationship graph, and extract temporal correlation information (such as role X - role Y: segment 1 stranger → segment 2 communication → segment 3 cooperation, etc.). Based on the temporal correlation information and combined with a preset relationship type library, the model can predict the temporal continuous relationship between the target role and other roles (excluding invalid types outside the library), and output multiple basic role relationship sequences (such as sequence 1: [stranger → communication → cooperation], sequence 2: [stranger → neutral → cooperation]).

[0141] Step 904: Call the role relationship quantification model and use the mapping table to process the basic role relationship graph to obtain the second value of each role relationship in the basic role relationship sequence under the preset quantification dimension, and weight the second value of the preset quantification dimension to obtain the second quantification index value sequence corresponding to the basic role relationship sequence.

[0142] Then, the role relationship quantification model can be called to process the basic role relationship graph using a mapping table, to obtain the second value of each role relationship in the basic role relationship sequence under the preset quantification dimension, and the second value of the preset quantification dimension can be weighted to obtain the second quantification index value sequence corresponding to the basic role relationship sequence.

[0143] Understandably, the method of obtaining the second quantitative indicator numerical sequence is similar to the method of obtaining the first quantitative indicator numerical sequence in step 901 above. The specific implementation process can be referred to the description of step 901 above, and will not be repeated here in this embodiment.

[0144] Step 905: Obtain the similarity between the first quantitative indicator value sequence and the second quantitative indicator value sequence.

[0145] After obtaining the first quantitative indicator numerical sequence and the second quantitative indicator numerical sequence, the similarity between the first quantitative indicator numerical sequence and the second quantitative indicator numerical sequence can be obtained.

[0146] In practical implementation, similarity calculation methods suitable for ordered numerical sequences can be used, such as cosine similarity (which measures the consistency of sequence direction) or Pearson correlation coefficient. The algorithm inputs the numerical sequences of the first and second quantification indicators to obtain the similarity score.

[0147] Step 906: Select target recommended video files from the video files to be recommended, whose similarity is greater than the similarity threshold.

[0148] The similarity threshold is a preset critical value (such as 0.8) for judging the similarity between two numerical sequences, used to filter out videos that meet preferences for recommendation.

[0149] After obtaining the similarity scores, target recommended video files with similarity scores greater than a similarity threshold can be selected from the video files to be recommended. Specifically, the similarity results of the videos to be recommended can be compared with the threshold, and the videos with similarity scores greater than the threshold can be identified as target recommended video files.

[0150] This application embodiment quantifies and compares the similarity of dynamic role relationships, which can accurately filter out videos to be recommended that are similar to the dynamic role relationships preferred by users. This realizes personalized video recommendation based on role relationship features, improving the accuracy of recommendations and user satisfaction.

[0151] Reference Figure 10 The flowchart illustrates the steps of a model training method provided in an embodiment of this application, as follows: Figure 10 As shown, the model training method may include steps 1001 to 1007.

[0152] Step 1001: Obtain the sample multimodal summary information of each segment of the sample video file. The sample multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file.

[0153] In this embodiment, the sample video file is video data (such as a complete TV series) used to train the dynamic role relationship prediction model, and is the basic material for the model to learn the evolution of role relationships.

[0154] The sample multimodal summary information is a comprehensive summary formed by integrating the role information (such as role identity and interaction status) and video dialogue (dialogue text) within a single segment of the sample video file. It is a multi-dimensional condensation of the core content of the segmented video file.

[0155] When training the role relationship prediction model, sample multimodal summary information of each segment of the sample video file can be obtained. The sample multimodal summary information is obtained by integrating the role information of each segment of the sample video file and the video screen dialogue of each segment.

[0156] The implementation process of step 1001 in this embodiment is similar to the implementation process of step 101 in the above scheme, and will not be described again in this embodiment.

[0157] Step 1002: Based on the sample multimodal summary information of all segmented video files of the sample video file, determine the character feature information of the target character in the sample video file; wherein, the character feature information includes: character introduction information and character experience information, wherein the character introduction information is the static attribute information of the target character extracted from the sample multimodal summary information, and the character experience information is the temporally continuous character event sequence of the target character extracted from the sample multimodal summary information.

[0158] Character characteristic information may include: character profile information and character experience information.

[0159] Among them, the character profile information can be the static attribute information of the target character extracted from the multimodal summary information of the sample. It can be the basic attributes of the target character (such as identity, personality, initial setting and other relatively stable features) extracted from the multimodal summary information of all episodes of the sample video.

[0160] Character experience information can be a temporally continuous sequence of character events extracted from sample multimodal summary information. It can be based on sample multimodal summary information of all episodes of the sample video, outlining the key development process of the target character in the entire series (important events, state changes, etc. arranged in episode order).

[0161] After obtaining the multimodal summary information of all segmented video files of the sample video file, the character feature information of the target character in the sample video file, namely the character introduction information and experience information of the target character, can be determined based on the multimodal summary information of all segmented video files of the sample video file.

[0162] Understandably, the implementation process of step 1002 in this embodiment is similar to that of step 102 described above, and will not be repeated here.

[0163] Step 1003: Input the character profile information, character experience information, and historical segmented character relationships of the character with a set number of segments into the character relationship generation model, and obtain the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located, as output by the character relationship generation model; wherein, the historical segmented character relationships refer to the segmented character relationships output by the model in the previous inference during the character relationship prediction process of the video file, and the historical segmented character relationships when the character relationship generation model first performs character relationship prediction on the video file are empty.

[0164] The segmented character relationships in the sample video refer to the specific associations (such as "cooperation" or "hostility") between the target character and other characters in each segment of the sample video file, which may change as the series progresses.

[0165] The target supporting event refers to the specific plot event used to prove the establishment of the relationship between the characters in the sample segment (such as "character A rescues character B" in a certain episode to support the "ally" relationship), which is the "evidence" of the relationship between the characters in the sample segment.

[0166] After obtaining the character feature information, an iterative input approach based on character relationships can be used. The character relationship generation model processes the character introduction and experience information, as well as the historical segmented character relationships, within the character feature information. This generates sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to these sample segmented character relationships, and the number of segmented files containing the target supporting events. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships in the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships in the video file. Understandably, the implementation process of step 1003 in this embodiment is similar to that of step 103 described above, and will not be repeated here.

[0167] Step 1004: Based on the sample segmented role relationships, the target supporting events, and the number of segmented files in which the target supporting events are located, construct a sample relationship graph of the target role, and label the real role relationship sequence corresponding to the sample relationship graph.

[0168] After obtaining the sample segment role relationships, target supporting events, and the number of segment files in which the target supporting events are located, a sample relationship graph of the target role can be constructed based on the sample segment role relationships, target supporting events, and the number of segment files in which the target supporting events are located, and the corresponding real role relationship sequence of the sample relationship graph can be labeled.

[0169] Step 1005: Input the sample relationship graph into the role relationship prediction model to be trained, and obtain the predicted role relationship sequence corresponding to the sample relationship graph output by the role relationship prediction model to be trained.

[0170] After constructing the sample relationship graph, the sample relationship graph can be input into the role relationship prediction model to be trained, and the predicted role relationship sequence corresponding to the sample relationship graph output by the role relationship prediction model to be trained can be obtained.

[0171] Step 1006: Calculate the loss value of the role relationship prediction model to be trained based on the real role relationship sequence and the predicted role relationship sequence.

[0172] After obtaining the predicted role relationship sequence, the loss value of the role relationship prediction model to be trained can be calculated based on the real role relationship sequence and the predicted role relationship sequence. Specifically, the difference (loss value) between the real role relationship sequence and the predicted role relationship sequence can be calculated by comparing the loss function (such as cross-entropy).

[0173] Step 1007: If the loss value indicates that the role relationship prediction model to be trained has converged, the trained role relationship prediction model is obtained.

[0174] After calculating the loss value, it can be determined whether the loss value is within a preset range. If the loss value indicates that the role relationship prediction model to be trained has converged, the trained role relationship prediction model is obtained.

[0175] Of course, if the loss value is not within the preset range, it means that the role relationship prediction model to be trained has not converged. In this case, the model parameters can be optimized through backpropagation to gradually reduce the loss value. When the loss value stabilizes at a low level (e.g., ≤0.05) and no longer decreases, the model is considered to have converged, and the trained role relationship prediction model is obtained.

[0176] The model training method provided in this application generates segmented role relationships of the characters to be analyzed by acquiring multimodal summary information of each segment of a video file, and constructs a fine-grained segmented role relationship graph. A role relationship prediction model is then trained based on this fine-grained segmented role relationship graph. This model can automatically generate dynamic role relationships in video files without manual intervention, thus achieving the goal of automatically generating dynamic role relationships. Furthermore, compared to manual annotation, this reduces labor costs.

[0177] Reference Figure 11 The diagram shows a structural schematic of a role relationship generation device provided in an embodiment of this application. Figure 11 As shown, the role relationship generation device 1100 may include the following modules: The summary acquisition module 1101 is used to acquire multimodal summary information of each segment of the video file. The multimodal summary information is obtained by integrating the character information and video dialogue in each segment of the video file. The feature determination module 1102 is used to determine the role feature information of the character to be analyzed in the video file based on the multimodal summary information of all segmented video files of the video file; The relationship generation module 1103 is used to process the role feature information using a role relationship generation model by adopting an iterative input method, and to generate segmented role relationships between the role to be analyzed and other roles in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located; the segmented role relationships represent the relationship between the role to be analyzed and other roles in the segmented video files, and the supporting events are events extracted from the segmented video files to characterize the relationship between the role to be analyzed and other roles; The graph construction module 1104 is used to construct a segmented role relationship graph of the role to be analyzed based on the segmented role relationship, the supporting events and the number of segmented files; The relationship sequence generation module 1105 is used to call a pre-trained role relationship prediction model to process the segmented role relationship graph and generate multiple temporally continuous role relationship sequences between the role to be analyzed and other roles in the video file.

[0178] Optionally, the summary acquisition module includes: The file segmentation unit is used to segment the video file to obtain segmented video files. The video frame acquisition unit is used to extract video frames from each of the segmented video files according to a first time interval to obtain the basic video frames of each of the segmented video files. The summary generation unit is used to generate multimodal summary information for each segmented video file based on the character information and video dialogue information in each of the basic video frames.

[0179] Optionally, the file segmentation unit includes: The transition determination subunit is used to determine the scene transition point of the video file based on the similarity between two adjacent video keyframes in the video file. The file segmentation subunit is used to segment the video file according to the scene turning point to obtain the segmented video file.

[0180] Optionally, the file segmentation unit includes: The episode acquisition subunit is used to acquire the video episodes of the video file; The segmented file acquisition subunit is used to segment the video file according to the video episode to obtain the segmented video file, and each segmented video file corresponds to one episode of the video file.

[0181] Optionally, the summary generation unit includes: The turning point determination subunit is used to determine the scene turning point of each segmented video file based on the similarity between two adjacent basic video frames in each segmented video file. The target frame acquisition subunit is used to extract video frames from the basic video frames between two adjacent scene transition points of each segmented video file according to a second time interval, so as to obtain the target video frames of all video scenes of each segmented video file; the video frames between two adjacent scene transition points constitute a video scene. The summary generation subunit is used to generate video summary information corresponding to each video scene based on the character information and video dialogue information in the target video frame of each video scene; The summary acquisition subunit is used to process the spliced ​​summary information of each segmented video file using a large language model to obtain the multimodal summary information of each segmented video file. The spliced ​​summary information is the summary information obtained by splicing the video summary information of all video scenes of each segmented video file.

[0182] Optionally, the inflection point determination subunit includes: A similarity acquisition subunit is used to acquire the similarity between two adjacent basic video frames of each segmented video file; The target video frame acquisition subunit is used to acquire two adjacent target base video frames whose similarity is less than the similarity threshold. The scene inflection point determination subunit is used to determine the previous target base video frame of the two adjacent target base video frames as the scene inflection point.

[0183] Optionally, the character feature information includes: character profile information and character experience information, wherein the character profile information is static attribute information of the character to be analyzed extracted from the multimodal summary information, and the character experience information is a temporally continuous sequence of character events of the character to be analyzed extracted from the multimodal summary information. The relationship generation module includes: An information input unit is used to input the character introduction information of a set number of segments, the character experience information, and the historical segmented character relationships into the character relationship generation model during each model inference process when performing relationship prediction using the character relationship generation model in the order of the segmented video files from front to back. The historical segmented character relationships refer to the segmented character relationships output by the model in the previous model inference process during the character relationship prediction of the video files. The historical segmented character relationships are empty when the character relationship generation model first performs character relationship prediction on the video files. The relationship acquisition unit is used to acquire the segmented role relationships obtained by the role relationship generation model through processing the role introduction information, the role experience information and the segmented role relationships output by the role relationship generation model last time, the supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located.

[0184] Optionally, the map construction module includes: The timing sequence determination unit is used to determine the timing sequence of the segment roles based on the segment order of the video file. The relationship determination unit is used to determine, based on the temporal order of the segmented role relationships, whether there is a relationship conflict between each segmented role relationship and the segmented role relationships of the N adjacent segmented video files; wherein, the relationship conflict refers to the situation where the change magnitude of the current segmented role relationship and the N adjacent segmented role relationships is greater than a threshold, and there are no supporting events representing the change of segmented role relationships among the supporting events of the current segmented role relationship and the N adjacent segmented role relationships, where N is a positive integer; The correction relationship acquisition unit is used to iteratively execute the relationship generation module if there is a relationship contradiction, until there is no relationship contradiction, and obtain the corrected segmented role relationship; The graph construction unit is used to construct a segmented role relationship graph of the role to be analyzed based on the corrected segmented role relationships, the supporting events, and the number of segmented files.

[0185] Optionally, the device further includes: The first numerical acquisition module is used to call the role relationship quantification model to process the segmented role relationship graph using a mapping table, obtain the first numerical value of each role relationship in the role relationship sequence in a preset quantification dimension, and weight the first numerical value of the preset quantification dimension to obtain the first quantification index numerical sequence corresponding to the role relationship sequence. The mapping table records the numerical value corresponding to the preset emotional quantification dimension, which includes at least one of the following: emotional dimension, trust dimension, and interaction dimension. The basic character relationship graph acquisition module is used to acquire the basic character relationship graph of the target characters in the video file to be recommended. The basic character relationship graph is constructed based on the segmented character relationships, supporting events, and number of segmented files of the target characters in the video file to be recommended. The basic sequence acquisition module is used to call the role relationship prediction model to extract the temporal association information between the target role and other roles in the video file to be recommended from the basic role relationship graph, and to perform temporal prediction on the temporal association information using a preset relationship type library to obtain multiple temporally continuous basic role relationship sequences between the target role and other roles in the video file to be recommended; wherein, the basic role relationship sequence includes: the role relationship type corresponding to each temporal segment; The second numerical acquisition module is used to call the role relationship quantification model to process the basic role relationship graph using the mapping table, obtain the second numerical value of each role relationship in the basic role relationship sequence under the preset quantification dimension, and weight the second numerical value of the preset quantification dimension to obtain the second quantification index numerical sequence corresponding to the basic role relationship sequence. A similarity acquisition module is used to acquire the similarity between the first quantization index numerical sequence and the second quantization index numerical sequence. The target file filtering module is used to filter out target recommended video files from the video files to be recommended, with a similarity greater than a similarity threshold.

[0186] The role relationship generation apparatus provided in this application generates segmented role relationships of the role to be analyzed by acquiring multimodal summary information of each segment of a video file, and constructs a fine-grained segmented role relationship graph. Based on a pre-trained dynamic role relationship prediction model, the fine-grained segmented role relationship graph is processed to generate dynamic role relationships. This process requires no manual intervention and achieves the goal of automatically generating dynamic role relationships. Furthermore, compared to manual annotation, it reduces labor costs.

[0187] Reference Figure 12The diagram shows a schematic representation of a model training device provided in an embodiment of this application. Figure 12 As shown, the model training device 1200 may include the following modules: The sample summary acquisition module 1201 is used to acquire sample multimodal summary information of each segment of the sample video file. The sample multimodal summary information is obtained by integrating the role information and video dialogue in each segment of the video file. The role feature determination module 1202 is used to determine the role feature information of the target role in the sample video file based on the sample multimodal summary information of all segmented video files of the sample video file; wherein, the role feature information includes: role introduction information and role experience information, wherein the role introduction information is the static attribute information of the target role extracted from the sample multimodal summary information, and the role experience information is the temporally continuous role event sequence of the target role extracted from the sample multimodal summary information; The segmented relationship acquisition module 1203 is used to input character introduction information, character experience information, and historical segmented character relationships into the character relationship generation model, which contains a set number of segments. It then acquires the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located, all output by the character relationship generation model. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships for the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships for the video file. The sample graph construction module 1204 is used to construct a sample relationship graph of the target role based on the sample segment role relationship, the target supporting event, and the number of segment files in which the target supporting event is located, and to label the real role relationship sequence corresponding to the sample relationship graph; The sample graph input module 1205 is used to input the sample relationship graph into the role relationship prediction model to be trained, and to obtain the predicted role relationship sequence corresponding to the sample relationship graph output by the relationship prediction model to be trained; The loss value calculation module 1206 is used to calculate the loss value of the role relationship prediction model to be trained based on the real role relationship sequence and the predicted role relationship sequence. The prediction model acquisition module 1207 is used to obtain the trained role relationship prediction model when the loss value indicates that the role relationship prediction model to be trained has converged.

[0188] The model training apparatus provided in this application generates segmented role relationships of the characters to be analyzed by acquiring multimodal summary information of each segment of a video file, and constructs a fine-grained segmented role relationship graph. A dynamic role relationship prediction model is then trained based on this fine-grained segmented role relationship graph. This model can automatically generate dynamic role relationships in video files without manual intervention, thus achieving the goal of automatically generating dynamic role relationships. Furthermore, compared to manual annotation, this reduces labor costs.

[0189] This application also provides an electronic device, such as... Figure 13 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. Memory 1303 is used to store computer programs; When processor 1301 executes a program stored in memory 1303, it performs the following steps: Obtain multimodal summary information for each segment of the video file, wherein the multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file; Based on the multimodal summary information of all segmented video files of the video file, determine the character feature information of the character to be analyzed in the video file; Using an iterative input approach based on role relationships, a role relationship generation model is used to process the role feature information, generating segmented role relationships between the role to be analyzed and other roles in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located. The segmented role relationships represent the relationship between the role to be analyzed and other roles in the segmented video files, and the supporting events are events extracted from the segmented video files to characterize the relationship between the role to be analyzed and other roles. Based on the segmented role relationships, the supporting events, and the number of segmented files, a segmented role relationship graph of the role to be analyzed is constructed. The segmented character relationship graph is processed by calling a pre-trained character relationship prediction model to generate a sequence of multiple temporally continuous character relationships between the character to be analyzed and other characters in the video file.

[0190] When the processor 1301 executes the program stored in the memory 1303, it performs the following steps: The multimodal summary information of each segment of the sample video file is obtained. The multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. Based on the multimodal summary information of all segmented video files of the sample video file, the character feature information of the target character in the sample video file is determined; wherein, the character feature information includes: character introduction information and character experience information, wherein, the character introduction information is the static attribute information of the target character extracted from the multimodal summary information of the sample video file, and the character experience information is the temporally continuous character event sequence of the target character extracted from the multimodal summary information of the sample video file; The character profile information, character experience information, and historical segmented character relationships of the specified number of segments are input into the character relationship generation model. The model then obtains the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships for the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships for the video file. Based on the sample segmented role relationships, the target supporting events, and the number of segmented files in which the target supporting events are located, a sample relationship graph of the target role is constructed, and the real role relationship sequence corresponding to the sample relationship graph is labeled. The sample relationship graph is input into the role relationship prediction model to be trained, and the predicted role relationship sequence corresponding to the sample relationship graph output by the relationship prediction model to be trained is obtained. The loss value of the role relationship prediction model to be trained is calculated based on the real role relationship sequence and the predicted role relationship sequence. If the loss value indicates that the role relationship prediction model to be trained has converged, the trained role relationship prediction model is obtained.

[0191] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0192] The communication interface is used for communication between the aforementioned terminal and other devices.

[0193] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0194] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0195] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the role relationship generation methods or model training methods described in the above embodiments.

[0196] In another embodiment provided in this application, a computer program product containing instructions is also provided, on which a computer program is stored, which, when run on a computer, causes the computer to execute any of the role relationship generation methods or model training methods described above.

[0197] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0198] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0199] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0200] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for generating role relationships, characterized in that, include: Obtain multimodal summary information for each segment of the video file, wherein the multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file; Based on the multimodal summary information of all segmented video files of the video file, determine the character feature information of the character to be analyzed in the video file; Using an iterative input approach based on role relationships, a role relationship generation model is used to process the role feature information, generating segmented role relationships between the role to be analyzed and other roles in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located; The segmented role relationship refers to the relationship between the role to be analyzed and other roles in the segmented video file, and the supporting events are events extracted from the segmented video file to characterize the relationship between the role to be analyzed and other roles; Based on the segmented role relationships, the supporting events, and the number of segmented files, a segmented role relationship graph of the role to be analyzed is constructed. The segmented character relationship graph is processed by calling a pre-trained character relationship prediction model to generate a sequence of multiple temporally continuous character relationships between the character to be analyzed and other characters in the video file.

2. The method according to claim 1, characterized in that, The process of obtaining multimodal summary information for each segment of the video file includes: The video file is segmented to obtain segmented video files. According to the first time interval, video frames are extracted from each segmented video file to obtain the basic video frames of each segmented video file; Based on the character information and video dialogue information in each of the basic video frames, multimodal summary information is generated for each of the segmented video files.

3. The method according to claim 2, characterized in that, The step of segmenting the video file to obtain segmented video files includes: The scene turning points of the video file are determined based on the similarity between two adjacent keyframes in the video file. The video file is segmented based on the scene transition points to obtain the segmented video file.

4. The method according to claim 2, characterized in that, The step of segmenting the video file to obtain segmented video files includes: Obtain the video episodes from the video file; The video file is segmented according to the video episode to obtain segmented video files, and each segmented video file corresponds to one episode of the video file.

5. The method according to claim 4, characterized in that, The step of generating multimodal summary information for each segmented video file based on the character information and video dialogue information in each of the basic video frames includes: The scene inflection points of each segmented video file are determined based on the similarity between two adjacent basic video frames in each segmented video file. According to the second time interval, video frames are extracted from the basic video frames between two adjacent scene transition points of each segmented video file to obtain the target video frames of all video scenes of each segmented video file; the video frames between two adjacent scene transition points constitute a video scene. Based on the character information and video dialogue information in the target video frame of each video scene, generate video summary information corresponding to each video scene; Using a large language model, the concatenated summary information of each segmented video file is processed to obtain the multimodal summary information of each segmented video file. The concatenated summary information is the summary information obtained by concatenating the video summary information of all video scenes of each segmented video file.

6. The method according to claim 5, characterized in that, The step of determining the scene transition point of each segmented video file based on the similarity between two adjacent basic video frames in each segmented video file includes: Obtain the similarity between two adjacent base video frames of each segmented video file; Obtain two adjacent target base video frames with a similarity less than a similarity threshold; The preceding target base video frame of the two adjacent target base video frames is determined as the scene inflection point.

7. The method according to claim 1, characterized in that, The character feature information includes: character profile information and character experience information, wherein the character profile information is static attribute information of the character to be analyzed extracted from the multimodal summary information, and the character experience information is a temporally continuous sequence of character events of the character to be analyzed extracted from the multimodal summary information. The method of using iterative input of role relationships involves processing the role feature information using a role relationship generation model to generate segmented role relationships between the role to be analyzed and other roles in the video file, supporting events corresponding to the segmented role relationships, and the number of segmented files in which the supporting events are located, including: During the process of using the character relationship generation model to predict relationships in the segmented video files from beginning to end, each time the model inference is performed, the character introduction information of the set number of segments, the character experience information, and the historical segmented character relationships are input into the character relationship generation model; wherein, the historical segmented character relationships refer to the segmented character relationships output by the previous model inference during the character relationship generation model's prediction of character relationships in the video files, and the historical segmented character relationships when the character relationship generation model first performs character relationship prediction on the video files are empty; The role relationship generation model is obtained by processing the role introduction information, the role experience information, and the segmented role relationship output by the previous role relationship generation model to obtain the segmented role relationship, the supporting events corresponding to the segmented role relationship, and the number of segmented files in which the supporting events are located.

8. The method according to claim 1, characterized in that, The step of constructing a segmented role relationship graph of the role to be analyzed based on the segmented role relationships, the supporting events, and the number of segmented files includes: Based on the sequential order of the video file segments, determine the temporal order of the segmented role relationships; Based on the temporal order of the segmented role relationships, determine whether there is a relationship conflict between each segmented role relationship and the segmented role relationships of the N adjacent segmented video files; wherein, the relationship conflict refers to the situation where the change magnitude of the current segmented role relationship and the N adjacent segmented role relationships is greater than a threshold, and there are no supporting events representing the change of segmented role relationships among the supporting events of the current segmented role relationship and the N adjacent segmented role relationships, where N is a positive integer; If there are contradictory relationships, the method of using the role relationship iterative input is executed iteratively. The role relationship generation model is used to process the role feature information and generate the segmented role relationship between the role to be analyzed and other roles in the video file, the supporting events corresponding to the segmented role relationship, and the number of segmented files in which the supporting events are located, until there are no contradictory relationships and the corrected segmented role relationship is obtained. Based on the corrected segmented role relationships, the supporting events, and the number of segmented files, a segmented role relationship graph of the role to be analyzed is constructed.

9. The method according to claim 1, characterized in that, After the pre-trained role relationship prediction model is invoked to process the segmented role relationship graph and generate multiple temporally consecutive role relationship sequences between the role to be analyzed and other roles in the video file, the method further includes: The role relationship quantification model is invoked to process the segmented role relationship graph using a mapping table, thereby obtaining the first value of each role relationship in the role relationship sequence in a preset quantification dimension. The first value of the preset quantification dimension is then weighted to obtain the first quantification index value sequence corresponding to the role relationship sequence. The mapping table records the values ​​corresponding to the preset emotional quantification dimension, which includes at least one of the following: emotional dimension, trust dimension, and interaction dimension. Obtain the basic role relationship graph of the target characters in the video files to be recommended. The basic role relationship graph is constructed based on the segmented role relationships, supporting events, and number of segmented files of the target characters in the video files to be recommended. The role relationship prediction model is invoked to extract the temporal association information between the target role and other roles in the video file to be recommended from the basic role relationship graph, and the temporal association information is temporally predicted using a preset relationship type library to obtain multiple temporally continuous basic role relationship sequences between the target role and other roles in the video file to be recommended; wherein, the basic role relationship sequence includes: the role relationship type corresponding to each temporal segment; The role relationship quantification model is invoked to process the basic role relationship graph using the mapping table, thereby obtaining the second value of each role relationship in the basic role relationship sequence under a preset quantification dimension. The second values ​​of the preset quantification dimension are then weighted to obtain the second quantification index value sequence corresponding to the basic role relationship sequence. Obtain the similarity between the first quantitative indicator numerical sequence and the second quantitative indicator numerical sequence; Select target recommended video files from the video files to be recommended, whose similarity is greater than the similarity threshold.

10. A model training method, characterized in that, include: The multimodal summary information of each segment of the sample video file is obtained. The multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. Based on the multimodal summary information of all segmented video files of the sample video file, the character feature information of the target character in the sample video file is determined; wherein, the character feature information includes: character introduction information and character experience information, wherein, the character introduction information is the static attribute information of the target character extracted from the multimodal summary information of the sample video file, and the character experience information is the temporally continuous character event sequence of the target character extracted from the multimodal summary information of the sample video file; The character profile information, character experience information, and historical segmented character relationships of the specified number of segments are input into the character relationship generation model. The model then obtains the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships for the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships for the video file. Based on the sample segmented role relationships, the target supporting events, and the number of segmented files in which the target supporting events are located, a sample relationship graph of the target role is constructed, and the real role relationship sequence corresponding to the sample relationship graph is labeled. The sample relationship graph is input into the role relationship prediction model to be trained, and the predicted role relationship sequence corresponding to the sample relationship graph output by the relationship prediction model to be trained is obtained. The loss value of the role relationship prediction model to be trained is calculated based on the real role relationship sequence and the predicted role relationship sequence. If the loss value indicates that the role relationship prediction model to be trained has converged, the trained role relationship prediction model is obtained.

11. A role relationship generation device, characterized in that, include: The summary acquisition module is used to acquire multimodal summary information of each segment of the video file. The multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. The feature determination module is used to determine the role feature information of the character to be analyzed in the video file based on the multimodal summary information of all segmented video files of the video file; The relationship generation module is used to process the character feature information using a character relationship generation model by adopting an iterative input method, and to generate segmented character relationships between the character to be analyzed and other characters in the video file, supporting events corresponding to the segmented character relationships, and the number of segmented files in which the supporting events are located; The segmented role relationship refers to the relationship between the role to be analyzed and other roles in the segmented video file, and the supporting events are events extracted from the segmented video file to characterize the relationship between the role to be analyzed and other roles; The graph construction module is used to construct a segmented role relationship graph of the role to be analyzed based on the segmented role relationships, the supporting events, and the number of segmented files. The relationship sequence generation module is used to call a pre-trained role relationship prediction model to process the segmented role relationship graph and generate multiple temporally continuous role relationship sequences between the role to be analyzed and other roles in the video file.

12. A model training device, characterized in that, include: The sample summary acquisition module is used to acquire sample multimodal summary information for each segment of the sample video file. The sample multimodal summary information is obtained by integrating the character information and video dialogue within each segment of the video file. The role feature determination module is used to determine the role feature information of the target role in the sample video file based on the sample multimodal summary information of all segmented video files of the sample video file; wherein, the role feature information includes: role introduction information and role experience information, wherein the role introduction information is static attribute information of the target role extracted from the sample multimodal summary information, and the role experience information is a temporally continuous sequence of role events of the target role extracted from the sample multimodal summary information; The segmented relationship acquisition module is used to input character introduction information, character experience information, and historical segmented character relationships into the character relationship generation model, which contains a set number of segments. It then acquires the sample segmented character relationships between the target character and other characters in the sample video file, the target supporting events corresponding to the sample segmented character relationships, and the number of segmented files in which the target supporting events are located, all output by the character relationship generation model. The historical segmented character relationships refer to the segmented character relationships output by the model during the previous inference process when the character relationship generation model predicts character relationships for the video file. The historical segmented character relationships are empty when the character relationship generation model first predicts character relationships for the video file. The sample graph construction module is used to construct a sample relationship graph of the target role based on the sample segment role relationship, the target supporting event, and the number of segment files in which the target supporting event is located, and to label the real role relationship sequence corresponding to the sample relationship graph; The sample relationship graph input module is used to input the sample relationship graph into the role relationship prediction model to be trained, and to obtain the predicted role relationship sequence corresponding to the sample relationship graph output by the relationship prediction model to be trained; The loss value calculation module is used to calculate the loss value of the role relationship prediction model to be trained based on the real role relationship sequence and the predicted role relationship sequence. The prediction model acquisition module is used to obtain the trained role relationship prediction model when the loss value indicates that the role relationship prediction model to be trained has converged.

13. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

15. A computer program product containing instructions, wherein a computer program is stored thereon, characterized in that, When the computer program is run on a computer, it causes the computer to perform the method as described in any one of claims 1-10.

Images