A method and apparatus for identifying social relationships in a video

Abstract

This application relates to a method and apparatus for identifying social relationships in videos. The method includes: clustering videos by role to obtain a set of video clips for each role; reorganizing the set of video clips for all roles based on overlapping video clips between different roles to obtain a set of social clips, wherein each social clip in the set of social clips contains at least two roles; inputting the set of social clips into a target recognition model, and identifying the social relationships between pairs of roles in the set of social clips based on features in the set of social clips, wherein each pair of roles includes two roles. This application can improve the effectiveness of social relationship recognition.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular to a method and apparatus for identifying social relationships in videos. Background Technology

[0002] Understanding social relationships within film and television scenes utilizes technological means and methods to analyze video footage, thereby extracting high-level semantic information to help people quickly understand social relationships within multimedia information. Social relationships refer to the relationships between people, such as father and son, superior and subordinate, husband and wife, etc. Identifying the social relationships between people in a scene helps promote the application of a wealth of commercial services, such as video content understanding and knowledge graph construction.

[0003] Existing social relationship recognition technologies mainly fall into two categories. One is image-based recognition, which requires the image to contain the person to be identified. The relationship between the people is determined by recognizing the features of the person and the scene. This is commonly seen in photos uploaded by users or group photos. However, in movies and TV shows, the scenes are constantly moving, and the relevant people rarely appear at the same time. Even if another person is blurred in the same scene, it is impossible to capture useful information.

[0004] Another type is video-based recognition. Existing technology can only output one relationship for a video clip, but in actual movies and TV shows, a single clip often presents multiple relationships, such as husband and wife, parent and child, resulting in poor social relationship recognition performance. Summary of the Invention

[0005] To address, or at least partially address, the aforementioned technical problems, this application provides a method and apparatus for identifying social relationships in videos.

[0006] In a first aspect, this application provides a method for identifying social relationships in videos, the method comprising:

[0007] By clustering the videos by character, a collection of video clips for each character is obtained;

[0008] Based on the video clips that overlap between different characters, the collection of video clips for all characters is reorganized to obtain a collection of social clips, wherein each social clip in the collection of social clips contains at least two characters;

[0009] The social fragment set is input into the target recognition model, and the social relationships between role pairs in the social fragment set are identified through the features in the social fragment set, wherein the role pair includes two roles.

[0010] Optionally, the set of social segments is input into a target recognition model, and the social relationships between the role pairs in the set of social segments are identified based on the features in the set of social segments, including:

[0011] Identify the roles within the aforementioned collection of social segments;

[0012] From the video, find the character image of each character in the collection of social clips, wherein the character image is obtained by character detection bounding boxes;

[0013] Input the character image and the set of social segments into the target recognition model;

[0014] The social relationships between the pairs of characters in the social segment collection are obtained by analyzing the features of the social segment collection and the character images using the target recognition model.

[0015] Optionally, the step of clustering videos by character to obtain a collection of video clips for each character includes:

[0016] By using a character clustering method, video frames of the same character in the same scene in the video are clustered to obtain a collection of video frames with multiple cluster centers, where each cluster center indicates a character.

[0017] Sort the video frame collections in descending order of the number of video frames they contain;

[0018] Select the first n video frames as the set of video clips featuring the main characters.

[0019] Optionally, before sorting the video frame set in descending order of the number of video frames it contains, the method further includes:

[0020] Calculate the similarity of the cluster centers in every two video frame sets;

[0021] If the similarity is greater than or equal to a preset threshold, then the cluster center of the two video frame sets is determined to be the same person.

[0022] The two video frames are combined into a single video frame set.

[0023] Optionally, selecting the set of the first n video frames as the set of video clips for the main character includes:

[0024] From a sorted set of multiple video frames, select the first n video frames.

[0025] Delete the target video frame set that does not meet the preset requirements in the first n video frame set, wherein the person in the target video frame set is blurred or the back view is obscured.

[0026] Select video frame sets according to the aforementioned sorting to complete the n video frame sets;

[0027] The complete set of n video frames is used as the set of video clips of the main character.

[0028] Optionally, based on the overlapping video clips between different characters, the collection of video clips for all characters is reorganized to obtain a collection of social clips, including:

[0029] Determine the start and end times of each video segment in the video clip collection;

[0030] Based on the start time and the end time, if there is an overlap between the video clips of the characters, then at least two video clips that have an overlap will be merged into a social clip, wherein the social clip contains at least two characters;

[0031] All social fragments are compiled into a social fragment collection.

[0032] Optionally, before inputting the set of social fragments into the target recognition model, the method further includes:

[0033] The video samples and social relationship tags are input into the initial recognition model, wherein each social segment sample in the video samples includes at least two roles;

[0034] The social segments are analyzed using the initial recognition model to output social relationship results;

[0035] If the cross-entropy loss function between the social relationship result and the social relationship label is greater than the loss threshold, then the gradient is backpropagated to the neural network parameters;

[0036] The target recognition model is obtained when the cross-entropy loss function is no greater than the loss threshold after multiple iterations.

[0037] Optionally, the features in the collection of social segments include visual features, text features, and voice features.

[0038] Secondly, a device for recognizing social relationships in videos is provided, the device comprising:

[0039] The clustering module is used to cluster videos by character to obtain a collection of video clips for each character;

[0040] The reorganization module is used to reorganize the video clip set of all characters based on the video clips that overlap between different characters, to obtain a social clip set, wherein each social clip in the social clip set contains at least two characters.

[0041] The identification module is used to input the social fragment set into the target identification model, and identify the social relationship between the role pairs in the social fragment set through the features in the social fragment set, wherein the role pair includes two roles.

[0042] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0043] Memory, used to store computer programs;

[0044] The processor, when executing a program stored in memory, implements the steps of any of the aforementioned methods for identifying social relationships in a video.

[0045] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the aforementioned methods for identifying social relationships in a video.

[0046] The technical solutions provided in this application have the following advantages compared with the prior art:

[0047] The method provided in this application involves a server acquiring a collection of video clips for each character. Based on overlapping video clips, it identifies social clips containing at least two characters, thus obtaining a collection of social clips. The server analyzes the collection of social clips using a target recognition model to determine the social relationships between each pair of characters. By analyzing social clips containing at least two characters, this application can obtain all social relationships in the video, improving the effectiveness of social relationship recognition. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating a method for identifying social relationships in a video, provided as an embodiment of this application;

[0051] Figure 2 A diagram illustrating social relationships provided in the embodiments of this application;

[0052] Figure 3 A schematic diagram of a social relationship recognition process in a video provided in this application embodiment.

[0053] Figure 4 A schematic diagram of the structure of a social relationship recognition device in a video provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0057] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a method for identifying social relationships in videos is provided.

[0058] This application provides a method for identifying social relationships in videos, which can be applied to a server to identify all social relationships in a video.

[0059] The following will describe in detail, with reference to specific implementation methods, a method for identifying social relationships in videos provided in this application, such as... Figure 1 As shown, the specific steps are as follows:

[0060] Step 101: By clustering the videos by character, a collection of video clips for each character is obtained.

[0061] In this embodiment of the application, a video includes multiple people, each playing a role, and these roles have social relationships, such as colleagues, spouses, father and daughter, etc., as detailed in the specific social relationships described below. Figure 2 As shown, Figure 2 This is an example diagram illustrating social relationships. The people depicted can be real people or characters with special effects, etc.

[0062] The server uses a role clustering method to cluster the roles in the video. For each role, the video clips in which that role appears are integrated together to obtain a collection of video clips for that role.

[0063] Step 102: Based on the overlapping video clips between different characters, reorganize the video clip collection of all characters to obtain the social clip collection.

[0064] Each social segment in the social segment collection contains at least two characters.

[0065] In this embodiment of the application, in a video clip, two, three, or even more characters may appear simultaneously in the same video frame, resulting in overlap between video clips of different characters. The server aggregates the video clips of all characters and reassembles them according to the overlapping video clips to obtain multiple social clips. Each social clip contains at least two characters, and the multiple social clips form a social clip set.

[0066] As an optional implementation, the social segment set is obtained by reorganizing the video segment set of all characters based on the overlapping video segments between different characters. This includes: determining the start time and end time of each video segment in the video segment set; merging at least two video segments with intersection based on the start time and end time into a social segment if there is an intersection between the video segments of the characters, wherein the social segment contains at least two characters; and constructing a social segment set from all the social segments.

[0067] In this embodiment, for each video segment set, the start and end times of each video segment are determined. Based on the duration and position of the video segments in the video, if there is overlap between video segments of different characters, indicating that at least two characters appear in the segment, then the at least two overlapping video segments are merged into a social segment, meaning that the social segment contains at least two characters. The server then constructs a social segment set from all social segments.

[0068] For example, the video clips of character A appear at times of [1-20, 40-50, 80-90] (in seconds), and the video clips of character B appear at times of [10-30, 70-100]. Since a social video clip must contain at least two characters, the clip at times of 40-50 will be discarded. The merged time will then be [1-30, 70-100], which is split into two clips, 1-30 and 70-100, based on the continuity of time.

[0069] Step 103: Input the social fragment set into the target recognition model, and identify the social relationships between the role pairs in the social fragment set through the features in the social fragment set.

[0070] The role pair consists of two roles.

[0071] In this embodiment, the server inputs a set of social segments into a target recognition model. The target recognition model analyzes the features in the set of social segments to identify the social relationships between each pair of roles. Each role pair includes two roles, thus identifying the social relationship between any two roles appearing in the same social segment. If a segment contains n roles, then the number of social relationships generated is... .

[0072] For example, if a social segment contains three roles {A, B, C}, then the role pairs are {AB, AC, BC}.

[0073] The features include visual features, text features, and voice features, and can be any feature in the video. This application does not impose specific restrictions on the features.

[0074] In this application, the server acquires a collection of video clips for each character. Based on overlapping video clips, it identifies social clips containing at least two characters, thus obtaining a collection of social clips. The server analyzes this collection of social clips using a target recognition model to determine the social relationships between each pair of characters. By analyzing social clips containing at least two characters, this application can obtain all social relationships in the video, improving the effectiveness of social relationship recognition.

[0075] As an optional implementation, obtaining a collection of video clips for each character by clustering the video into characters includes: using a character clustering method, clustering video frames of the same character in the same scene in the video to obtain a collection of video frames with multiple cluster centers, wherein each cluster center indicates a character; sorting the video frame collections in descending order of the number of video frames they contain; and selecting the top n video frame collections as the collection of video clips for the main characters.

[0076] In this embodiment of the application, the server uses any role clustering method to cluster video frames of the same person in the same scene in the video, and obtains a collection of video frames for each cluster center, with each cluster center indicating a person.

[0077] Because people are at different distances and angles in different video frames, the same person might be assigned to multiple cluster centers, leading to inaccurate clustering. Therefore, the server calculates the similarity between every two cluster centers. If the similarity is less than a preset threshold, the two cluster centers are considered to belong to different people; if the similarity is greater than or equal to the preset threshold, the two cluster centers are considered to belong to the same person, and the two video frames corresponding to those two cluster centers are merged into a single video frame set. This application uses cluster center similarity filtering to ensure that each cluster center points to one person, thereby reducing the computational load in subsequent video segment reassembly.

[0078] The server sorts the video frame sets from largest to smallest according to the number of video frames contained. The earlier the video frame set appears, the more video frames a character has, indicating that the character is a main character. Therefore, the first n video frame sets are selected as the video clip sets for the main characters.

[0079] When the main character is the subject of the video clip collection, any step after character clustering can be considered as an operation step on the video clip of the main character. This avoids taking into account unimportant people, such as background passersby, reduces the identification of unimportant social relationships, and improves identification efficiency.

[0080] As an optional implementation, selecting the first n video frames as the video clip set for the main character includes: selecting the first n video frames from a sorted set of multiple video frames; deleting target video frames from the first n video frames that do not meet preset requirements, wherein the characters in the target video frames are blurred or their backs are obscured; selecting video frames in sorted order to complete the set of n video frames; and using the completed set of n video frames as the video clip set for the main character.

[0081] The server first selects the first n video frame sets from the sorted set of multiple video frames, and then determines whether there is a target video frame set in the n video frame sets that does not meet the preset requirements. The preset requirements are not met because the people in the video frame set are blurry or their backs are obscured. Since it is impossible to determine the relationship between people in such a picture, such video frames will be deleted, which helps to improve the accuracy of tagging.

[0082] If it is determined that the set of n video frames does not exist, then the set of n video frames is directly used as the set of video clips of the main character; if it is determined that the set of n video frames does not meet the preset requirements, then the set of n video frames is selected according to the order of the set of video frames to complete the set of n video frames. Finally, the set of n video frames is used as the set of video clips of the main character.

[0083] As an optional implementation, the social segment collection is input into the target recognition model, and the social relationships between role pairs in the social segment collection are identified by the features in the social segment collection. This includes: determining the roles in the social segment collection; finding the role image of each role in the social segment collection from the video, wherein the role image is obtained by person detection bounding boxes; inputting the role image and the social segment collection into the target recognition model; and analyzing the features of the social segment collection and the role image through the target recognition model to obtain the social relationships between role pairs in the social segment collection.

[0084] In this embodiment, before clustering characters, the server first detects the coordinates of character bounding boxes in the video to obtain character detection boxes in each video frame. After obtaining the social segment collection, the server determines each character in the collection, and then extracts a character image of each character from the video based on the character detection boxes. The character image contains only that character and does not contain other characters or excessive background elements.

[0085] The server inputs the set of character images and social segments into the target recognition model. The target recognition model analyzes the features of the social segment set and the features of the character images to obtain the social relationships between character pairs in the social segment set.

[0086] This application incorporates character images for entity analysis and combines them with complete video features to enrich feature information and improve the recognition of social relationships.

[0087] As an optional implementation, before inputting the social segment set into the target recognition model, the method further includes: inputting video samples and social relationship tags into an initial recognition model, wherein each social segment sample in the video sample includes at least two roles; analyzing the social segments through the initial recognition model and outputting social relationship results; if the cross-entropy loss function between the social relationship results and the social relationship tags is greater than a loss threshold, then backpropagating the gradient to the neural network parameters; after multiple iterations until the cross-entropy loss function is no greater than the loss threshold, the target recognition model is obtained.

[0088] The server selects a set of film and television videos V = {vi} as the training set. For each video, it performs character clustering, sorts the videos by the number of video frames corresponding to each cluster center, and selects the top n cluster centers as the main characters C = {Cvi-0, Cvi-1, ..., Cvi-n}, thus obtaining video samples of the main characters. The method for obtaining these video samples is the same as that for obtaining the video clip set, both using character clustering and sorting by the number of video frames.

[0089] The server obtains social relationship labels between characters in the video samples, uses the labeled relationships as label supervision information, and then inputs the video samples and social relationship labels into the initial recognition model. The initial recognition model analyzes the social segments and outputs the social relationship results. If the cross-entropy loss function between the social relationship results and the social relationship labels is greater than the loss threshold, the gradient is backpropagated to the neural network parameters. After multiple iterations, when the cross-entropy loss function is no greater than the loss threshold, the target recognition model is obtained.

[0090] Based on the same technical concept, this application also provides a schematic diagram of a social relationship recognition process in a video, such as... Figure 3 As shown, taking a social video clip featuring two characters as an example, character image 1 and character image 2 are obtained through character detection boxes. Character clustering yields a set of video clips for character 1 and a set of video clips for character 2. These two video clip sets are then reorganized to obtain a set of social video clips containing both characters 1 and 2. The character images are input into the average pooling layer of the target recognition model, the set of video clips is input into the max pooling layer, and then into a fully connected layer (FC layer) to obtain the recognition result.

[0091] Based on the same technical concept, embodiments of this application also provide a device for recognizing social relationships in videos, such as... Figure 4 As shown, the device includes:

[0092] Clustering module 401 is used to cluster videos by character to obtain a collection of video clips for each character;

[0093] The reorganization module 402 is used to reorganize the video clip set of all characters based on the video clips that overlap between different characters, to obtain a social clip set, wherein each social clip in the social clip set contains at least two characters.

[0094] The recognition module 403 is used to input the social fragment set into the target recognition model and identify the social relationship between the role pairs in the social fragment set through the features in the social fragment set, wherein the role pair includes two roles.

[0095] Optionally, the identification module 403 is used for:

[0096] Determine the roles within the social media clip collection;

[0097] Find the character image of each character in the collection of social clips from the video, where the character image is obtained by character detection bounding boxes;

[0098] Input the combined set of character images and social media clips into the target recognition model;

[0099] By analyzing the features of social fragment sets and character images using a target recognition model, the social relationships between character pairs within the social fragment sets can be obtained.

[0100] Optionally, clustering module 401 is used for:

[0101] By using the character clustering method, video frames of the same character in the same scene are clustered to obtain a collection of video frames with multiple cluster centers, where each cluster center indicates a character.

[0102] Sort the video frame collections in descending order of the number of video frames they contain;

[0103] Select the first n video frames as the set of video clips featuring the main characters.

[0104] Optionally, the device is also used for:

[0105] Calculate the similarity of the cluster centers in every two video frame sets;

[0106] If the similarity is greater than or equal to the preset threshold, then the cluster center of the two video frame sets is determined to be the same person.

[0107] Combine two video frames into a single video frame set.

[0108] Optionally, clustering module 401 is used for:

[0109] From a sorted set of multiple video frames, select the first n video frames.

[0110] Delete the target video frame set that does not meet the preset requirements from the first n video frame sets, where the people in the target video frame set are blurred or their backs are obscured.

[0111] Select video frame sets according to the order to complete the set of n video frames;

[0112] The complete set of n video frames is used as the set of video clips of the main character.

[0113] Optionally, the reforming module 402 is used for:

[0114] Determine the start and end times of each video segment in the video clip collection;

[0115] Based on the start and end times, if there is overlap between video clips of characters, at least two overlapping video clips will be merged into a social clip, wherein the social clip contains at least two characters.

[0116] All social fragments are compiled into a social fragment collection.

[0117] Optionally, the device is also used for:

[0118] Input video samples and social relationship labels into the initial recognition model, wherein each social segment sample in the video sample includes at least two roles;

[0119] The social segments are analyzed using an initial identification model, and the social relationship results are output.

[0120] If the cross-entropy loss function between the social relationship result and the social relationship label is greater than the loss threshold, then the gradient is backpropagated to the neural network parameters;

[0121] When the cross-entropy loss function is no greater than the loss threshold after multiple iterations, the target recognition model is obtained.

[0122] Optionally, the features in the social fragment set include visual features, text features, and voice features.

[0123] Based on the same technical concept, embodiments of the present invention also provide an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0124] Memory 503 is used to store computer programs;

[0125] The processor 501 performs the above steps when executing the program stored in the memory 503.

[0126] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0127] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0128] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0129] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0130] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program, when executed by a processor, implements the steps of any of the above methods.

[0131] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described above.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0133] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for identifying social relationships in videos, characterized in that, The method includes: By clustering the videos by character, a collection of video clips for each character is obtained, wherein the videos are film and television videos; Based on the video clips that overlap between different characters, the collection of video clips for all characters is reorganized to obtain a collection of social clips, wherein each social clip in the collection of social clips contains at least two characters; The social fragment set is input into the target recognition model, and the social relationships between role pairs in the social fragment set are identified through the features in the social fragment set, wherein the role pair includes two roles; Specifically, the social segment set is input into the target recognition model, and the social relationships between the role pairs in the social segment set are identified through the features in the social segment set, including: Identify the roles within the aforementioned collection of social segments; From the video, find the character image of each character in the collection of social clips, wherein the character image is obtained by character detection bounding boxes; The character image is input into the average pooling layer of the target recognition model, the social fragment set is input into the max pooling layer of the target recognition model, and then the processed character image and social fragment set are input into the fully connected layer. The target recognition model analyzes the features of the social segment set and the character image to obtain the social relationships between character pairs in the social segment set. The features in the social segment set include image features, text features, and voice features, and the character image is used for character entity analysis.

2. The method according to claim 1, characterized in that, The process of clustering videos by character to obtain a collection of video clips for each character includes: By using a character clustering method, video frames of the same character in the same scene in the video are clustered to obtain a collection of video frames with multiple cluster centers, where each cluster center indicates a character. Sort the video frame collections in descending order of the number of video frames they contain; Select the first n video frames as the set of video clips featuring the main characters.

3. The method according to claim 2, characterized in that, Before sorting the video frame collections in descending order of the number of video frames they contain, the method further includes: Calculate the similarity of the cluster centers in every two video frame sets; If the similarity is greater than or equal to a preset threshold, then the cluster center of the two video frame sets is determined to be the same person. The two video frames are combined into a single video frame set.

4. The method according to claim 2, characterized in that, The set of the first n video frames selected as the video clips featuring the main characters includes: From a sorted set of multiple video frames, select the first n video frames. Delete the target video frame set that does not meet the preset requirements in the first n video frame set, wherein the person in the target video frame set is blurred or the back view is obscured. Select video frame sets according to the aforementioned sorting to complete the n video frame sets; The complete set of n video frames is used as the set of video clips of the main character.

5. The method according to claim 1, characterized in that, Based on the overlapping video clips between different characters, the collection of video clips for all characters is reorganized to obtain the social clip collection, which includes: Determine the start and end times of each video segment in the video clip collection; Based on the start time and the end time, if there is an overlap between the video clips of the characters, then at least two video clips that have an overlap will be merged into a social clip, wherein the social clip contains at least two characters; All social fragments are compiled into a social fragment collection.

6. The method according to claim 1, characterized in that, Before inputting the set of social fragments into the target recognition model, the method further includes: The video samples and social relationship tags are input into the initial recognition model, wherein each social segment sample in the video samples includes at least two roles; The social segments are analyzed using the initial recognition model to output social relationship results; If the cross-entropy loss function between the social relationship result and the social relationship label is greater than the loss threshold, then the gradient is backpropagated to the neural network parameters; The target recognition model is obtained when the cross-entropy loss function is no greater than the loss threshold after multiple iterations.

7. A social relationship recognition device in a video, characterized in that, The device includes: The clustering module is used to cluster videos by character to obtain a collection of video clips for each character, wherein the videos are film and television videos; The reorganization module is used to reorganize the video clip set of all characters based on the video clips that overlap between different characters, to obtain a social clip set, wherein each social clip in the social clip set contains at least two characters. The identification module is used to input the social fragment set into the target identification model, and identify the social relationship between the role pairs in the social fragment set through the features in the social fragment set, wherein the role pair includes two roles; The identification module is used for: Identify the roles within the aforementioned collection of social segments; From the video, find the character image of each character in the collection of social clips, wherein the character image is obtained by character detection bounding boxes; The character image is input into the average pooling layer of the target recognition model, the social fragment set is input into the max pooling layer of the target recognition model, and then the processed character image and social fragment set are input into the fully connected layer. The target recognition model analyzes the features of the social segment set and the character image to obtain the social relationships between character pairs in the social segment set. The features in the social segment set include image features, text features, and voice features, and the character image is used for character entity analysis.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images