Method, apparatus, device and storage medium for determining a video segment

Abstract

Embodiments of the present disclosure provide a method, device, equipment, storage medium and program product for determining a video clip. The method comprises obtaining a target video and description information associated with a target user, the target video comprising video frame information, audio information and subtitle information, the description information indicating requirements related to clip extraction of the target video; extracting a visual feature representation from the video frame information, extracting an audio feature representation from the audio information, extracting a first text feature representation from the subtitle information, and extracting a second text feature representation from the description information; determining at least one candidate video clip matching the description information from the target video based on the visual feature representation, the audio feature representation, the first text feature representation and the second text feature representation; and determining at least one target video clip related to the target user from the at least one candidate video clip based on at least the respective subtitle information of the at least one candidate video clip.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, devices, and computer-readable storage media for determining video segments. Background Technology

[0002] With the development of information technology, video, as a vivid means of communication, has permeated all aspects of human life. However, as video content becomes increasingly abundant, viewers' time has become increasingly precious. Faced with lengthy videos, how to quickly grasp their essence has become a common challenge for both viewers and creators. Therefore, extracting highlight segments from long videos is particularly important. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for determining video segments is provided. The method includes: acquiring a target video and descriptive information associated with a target user, the target video including video frame information, audio information, and subtitle information, the descriptive information indicating requirements related to segment extraction from the target video; extracting visual feature representations from the video frame information, audio feature representations from the audio information, a first text feature representation from the subtitle information, and a second text feature representation from the descriptive information; determining at least one candidate video segment from the target video that matches the descriptive information based on the visual feature representations, audio feature representations, the first text feature representation, and the second text feature representation; and determining at least one target video segment associated with the target user from at least one candidate video segment based at least on the subtitle information corresponding to each of the at least one candidate video segment.

[0004] In a second aspect of this disclosure, an apparatus for determining video segments is provided. The apparatus includes: an information acquisition module configured to acquire a target video and descriptive information associated with a target user, the target video including video frame information, audio information, and subtitle information, the descriptive information indicating requirements related to segment extraction from the target video; a feature extraction module configured to extract visual feature representations from the video frame information, audio feature representations from the audio information, a first text feature representation from the subtitle information, and a second text feature representation from the descriptive information; a candidate video segment determination module configured to determine at least one candidate video segment from the target video that matches the descriptive information based on the visual feature representations, audio feature representations, the first text feature representation, and the second text feature representation; and a target video segment determination module configured to determine at least one target video segment associated with the target user from at least one candidate video segment based at least on subtitle information corresponding to each of the at least one candidate video segment.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram illustrating the process of determining video segments according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating the determination of at least one candidate video segment using a multimodal transformer model according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrates an example process for determining a target video segment from candidate video segments according to some embodiments of the present disclosure;

[0014] Figure 5 A schematic diagram illustrates an example process for determining the identification information of a target user according to some embodiments of the present disclosure;

[0015] Figure 6 A schematic diagram illustrating an example process for determining a second set of video segments according to some embodiments of the present disclosure is shown;

[0016] Figure 7A flowchart of a method for determining video segments according to some embodiments of the present disclosure is shown;

[0017] Figure 8 An apparatus for determining video segments according to some embodiments of the present disclosure is shown; and

[0018] Figure 9 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0023] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0027] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0028] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0029] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, a segment extraction system 120 is deployed in an electronic device 110. The segment extraction system 120 is configured to determine a plurality of video segments 104-1 to 104-N (which may be collectively referred to as video segments 104 for ease of description) in a target video 102.

[0030] In some embodiments, target video 102 may include videos such as movies, news reports, sports events, and game videos. Video clip 104 may be a segment of the overall content of target video 102 that is considered relatively important, also known as a "highlight" segment. These segments may be video clips in target video 102 that contain a lot of information or leave a deep impression on the viewer.

[0031] In some embodiments, the segment extraction system 120 can extract video segments 104 from the target video 102 based on a machine learning model.

[0032] In environment 100, electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] As mentioned earlier, extracting highlight segments from long videos is crucial. For videos related to interactive events (e.g., interactive videos, live streams, game videos), sharing users' impressive actions during these events is a major demand on content platforms. For example, in gaming scenarios, screen recordings are an important way to capture these highlights. However, pure screen recordings have low information density, lacking highlights and focus, which is not conducive to the rapid dissemination of content.

[0035] One approach to generating highlight segments in videos is to utilize anchor point information embedded within applications (such as live streaming apps and gaming apps) to locate specific highlight segments. However, obtaining anchor point information within applications is difficult and costly for content platforms, failing to meet users' requirements for extracting highlight segments.

[0036] Another approach is to train a powerful machine learning model to cover all applications, using the model to retrieve highlight clips and automatically synthesize highlight reels. However, because the machine learning model needs to cover as many application types as possible, the model training objectives need to be as uniform as possible, thus only producing relatively simple highlight content. Furthermore, since the definition of a highlight is subjective, simple highlight content cannot satisfy all users' preferences, making it impossible to produce more customized highlight videos. Therefore, it is clear that the relevant technical solutions all have certain limitations in generating highlight clips.

[0037] In embodiments of this disclosure, a method for determining video segments is proposed. Specifically, a target video and descriptive information associated with a target user are acquired. The target video includes video frame information, audio information, and subtitle information, and the descriptive information indicates requirements related to segment extraction from the target video. Visual feature representations are extracted from the video frame information, audio feature representations are extracted from the audio information, a first text feature representation is extracted from the subtitle information, and a second text feature representation is extracted from the descriptive information. Based on the visual feature representations, audio feature representations, the first text feature representation, and the second text feature representation, at least one candidate video segment matching the descriptive information is determined from the target video. At least one target video segment associated with the target user is determined from at least one candidate video segment, based at least on the subtitle information corresponding to each of the at least one candidate video segment.

[0038] According to the scheme disclosed herein, by utilizing multimodal information from the target video and descriptive information, the content included in the target video can be comprehensively understood, and the descriptive information can be used to determine candidate video segments that match the segment extraction requirements of the target user. Furthermore, subtitle information in the candidate video segments can filter out video segments irrelevant to the target user. In this way, the efficiency and accuracy of extracting target video segments relevant to the target user from the target video can be improved.

[0039] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0040] Figure 2 A schematic diagram of a process 200 for determining video segments according to some embodiments of the present disclosure is shown. Figure 2 As shown, the process first acquires the target video 102 and descriptive information 202 associated with the target user. The target video 102 includes video frame information, audio information, and subtitle information. The descriptive information 202 indicates the requirements related to segment extraction from the target video 102. In one example, the target video 102 is a movie, and the descriptive information 202 could indicate extracting segments from the movie that highlight its essence and leave a lasting impression on the audience. In another example, the target video 102 is a game video, and the descriptive information 202 could indicate extracting segments from the game video showcasing impressive maneuvers performed by the target player.

[0041] In some embodiments, video frames can be extracted from the target video 102 at a frequency corresponding to a predetermined frame rate, such as 1 FPS (frames per second), and audio information aligned with the video frames can also be extracted. Each extracted video frame can be considered an image. For each extracted frame, an optical character recognition (OCR) model can be used to extract caption information from the frame, which may include text information and a bounding box indicating the location of the text information.

[0042] In box 204, multimodal preprocessing operations can be performed on the target video 102 and the description information 202. Specifically, visual feature representations are extracted from video frame information, audio feature representations are extracted from audio information, a first text feature representation is extracted from subtitle information, and a second text feature representation is extracted from description information.

[0043] In some embodiments, different feature encoders can be used to extract feature vectors (also called feature representations) from information of different modalities. Visual feature representations can be extracted using a visual encoder, audio feature representations can be extracted using an audio encoder, and first text feature representations and second text feature representations can be extracted using a text encoder.

[0044] In box 206, at least one candidate video segment 208 matching the descriptive information can be determined from the target video 102 based on visual feature representation, audio feature representation, a first text feature representation, and a second text feature representation. In some embodiments, the at least one candidate video segment 208 can be determined using a multimodal transformer model. According to embodiments of this disclosure, multimodal data enables the multimodal transformer model to more comprehensively understand the content of the target video 102, and multimodal data can provide additional contextual information, helping the multimodal transformer model extract candidate video segments that meet the needs of the target user from the target video 102. In this way, the efficiency and accuracy of extracting target video segments relevant to the target user from the target video can be improved. Reference will be made below. Figure 3 Describe the process of determining at least one candidate video segment 208 using a multimodal transformer model.

[0045] In box 210, at least one target video segment 212 related to the target user is determined from at least one candidate video segment 208, based at least on the caption information corresponding to each of the at least one candidate video segment. In some embodiments, the at least one target video segment 212 may be merged into a complete video, and the complete video may be provided to the target user.

[0046] In some embodiments, for each of the at least one candidate video segments, a candidate user identifier associated with the caption information included in the candidate video segment can be determined. It can then be determined whether the candidate user identifier matches the identifier information of a target user. In response to a match, the candidate video segment is identified as target video segment 212. If the candidate user identifier matches the identifier information of the target user, it indicates that the candidate video segment is related to the target user, and therefore the candidate video segment can be identified as target video segment 212. In some examples, the candidate user identifier associated with the caption information may include a username, avatar, etc. In one example, the caption information is information about a user entering a live stream, in which case the candidate user identifier associated with the caption information could be a username. In another example, the caption information is game status information in a game, in which case the candidate user identifier associated with the caption information could be a user avatar. According to embodiments of this disclosure, target video segments related to target users can be accurately determined based on video-specific caption information and identifier information related to interactive events. In this way, the accuracy and recall rate of target video segment generation can be improved.

[0047] In a candidate video clip, a candidate user identifier typically appears at a specific location at a specific time. Based on the visual style of the candidate user identifier or its similarity to other known user identifiers, it can be determined whether the candidate user identifier is a user identifier for the target user.

[0048] In some embodiments, a region containing subtitle information can be determined from candidate video segments. Then, identification information located at a predetermined position within that region is identified as a candidate user identifier. The subtitle information may include the current actions of various players in the game, and the identifier information of the player involved in the current action will be displayed at a specific position in the subtitle information. Therefore, candidate user identifiers can be extracted based on the region containing subtitle information. In some examples, the region containing subtitle information can be a bounding box. For example, game status information in the game can be included in a bounding box, and identification information located at a predetermined position within that bounding box (e.g., left or right side) can be identified as a candidate user identifier. It should be noted that determining candidate user identifiers through a region containing subtitle information is only an example, and this disclosure does not limit the method for determining candidate user identifiers. The method for extracting user identifiers can be configured according to the fixed patterns in which user identifiers may appear in a specific application scenario.

[0049] In game scenarios, depending on some game settings, user identifiers (e.g., user avatars) can be visually distinguished from opposing team members using their visual style (e.g., border color). For example, a user identifier with a blue border indicates that the user is an allied member, while a user identifier with a red border indicates that the user is an opposing member. After determining that a user identifier belongs to an allied member, it is necessary to determine whether that user identifier belongs to the target user themselves or to a teammate. Typically, teammate identifiers are displayed in a specific location in the game video. By comparing the similarity between the target user identifier and the teammate's user identifier, it can be determined whether the user identifier belongs to the player or a teammate.

[0050] In some embodiments, it can be determined whether a candidate user identifier matches the identifier information of the target user based on the visual style of the candidate user identifier and the similarity between the candidate user identifier and the identifier information of the interactive user corresponding to the target user. The visual style of the candidate user identifier may include the color or shape of the border of the candidate user identifier. For example, if the border color of the candidate user identifier is a specific color (e.g., blue), it can be determined that the candidate user identifier is an allied user identifier. Then, it can be determined whether the candidate user identifier matches the identifier information of the target user based on the similarity between the candidate user identifier and the identifier information of the interactive user corresponding to the target user. In a live streaming scenario, the interactive user corresponding to the target user may include users who are connected to the target user (e.g., streamers). In a game scenario, the interactive user corresponding to the target user may include the target user's teammates. In some examples, the identifier information of the interactive user corresponding to the target user may be obtained in advance. If the similarity between the candidate user identifier and the identifier information of the interactive user is greater than a predetermined threshold (e.g., 60%), it is said that the candidate user identifier does not match the identifier information of the target user. If the similarity between the candidate user identifier and the identifier information of the interactive user is less than the predetermined threshold, it is said that the candidate user identifier matches the identifier information of the target user. This method allows for the filtering of video clips irrelevant to the target user, thereby improving the accuracy of generating the target video clips. The following section will refer to... Figure 4 Describe an example process for determining the target video segment 212 from candidate video segment 208.

[0051] In addition to the methods described in boxes 204 to 210 above, which utilize a multimodal transformer model to identify at least one candidate video segment and determine the target video segment 212 from at least one candidate video segment based on subtitle information, the target video segment 212 related to the target user can also be filtered directly from the target video 102 using the target user's user identifier. Therefore, it is necessary to first extract the target user's identifier information.

[0052] In box 214, the target user's identification information can be extracted. In some examples, the target user's identifier can be user-specified, or it can be determined automatically from video related to the interaction event.

[0053] In some embodiments, a reference video segment (also referred to as a first reference video segment) including the identification information of the target user can be determined from the video associated with the interactive event. This reference video segment has predetermined caption information or is associated with an action performed by the target user during the interactive event. The target user's identification information can then be extracted from the reference video segment. In some examples, the reference video segment has predetermined caption information that may indicate that the reference video segment includes the target user's identification information. For example, the predetermined caption information may include "You have entered the live stream" or "You have been defeated by the opponent." In some examples, the action performed by the target user during the interactive event may indicate that the first reference video segment includes the target user's identification information. Taking a game video as an example, the action performed by the target user during the interactive event may include specific skills that the target user's controlled character can unleash. References will follow below. Figure 5 An example process for determining the identification information of a target user.

[0054] In box 216, if the target user's identification information is successfully extracted, process 200 proceeds to box 218. In box 218, the target user's identification information determined from box 214 can be used to recall video segments including the target user's identification information, thus identifying target video segment 212. In box 220, the recalled target video segment 212 can be deduplicated with the target video segment 212 determined by the operation in box 210, thereby identifying all target video segments.

[0055] In some embodiments, the target video segment 212 determined by the operations in block 210 may be referred to as the first group of video segments. In some examples, at least one candidate video segment may be determined using a multimodal converter modality, and the first group of video segments may be determined based on the subtitle information corresponding to each of the at least one candidate video segment.

[0056] In some embodiments, the target video segment 212 determined by the operation in block 218 may be referred to as the second set of video segments. In some examples, the second set of video segments containing the target user's identification information may be determined from the target video 102 based on the target user's identification information.

[0057] In some embodiments, to implement the operation of recalling target video segment 212 in block 218, at least one reference video segment (also referred to as a second reference video segment) can be determined from the target video 102, each reference video segment having caption information that matches the description information 202. For each of the at least one reference video segment, it can be determined whether the candidate user identifier associated with the caption information matches the identifier information of the target user. If the candidate identifier information matches the identifier information of the target user, the reference video segment is determined as a video segment in the second set of target video segments.

[0058] After identifying the second set of video clips, at least one target video clip can be determined by merging the first and second sets of video clips. The merging of the first and second sets of video clips is primarily for the purpose of deduplicating the video clips.

[0059] In some embodiments, video segments in the second set of video segments that overlap with those in the first set of target video segments can be filtered out, and the filtered second set of video segments and the first set of target video segments can be jointly determined as the final target video segments relevant to the target user. References will follow below. Figure 6 Describe an example process for recalling the second set of video clips.

[0060] The following will continue to refer to the appendix Figure 3-6 Example procedures are described according to some example embodiments of this disclosure.

[0061] Figure 3 A schematic diagram of a process 300 for determining at least one candidate video segment 208 using a multimodal transformer model according to some embodiments of the present disclosure is shown. Figure 3 As shown, different encoders can be used to encode information of different modalities (also known as feature extraction). First, a visual feature representation 313 can be extracted from video frame information 302 using a visual encoder 308. In some embodiments, a visual encoder (e.g., a ViT-B visual encoder) of a contrastive language-image pre-trained (CLIP) model pre-trained in the domain of the target video 102 can be used to extract primary features from video frame information 302, the primary features having a dimension of 768. Furthermore, to enhance the scalability of the multimodal transformer model 318 and improve the recognition ability of general highlight segments, auxiliary features can be extracted using the ViT-B visual encoder of the original CLIP model, the auxiliary features having a dimension of 512. Then, the primary and auxiliary features are concatenated to obtain a visual feature representation 313 with a dimension of (N, 1280), where N is the total number of video frames in the target video 102.

[0062] Then, an audio feature representation 314 can be extracted from the audio information 304 using an audio encoder 310. In some embodiments, the audio information 304 can be encoded using an audio feature encoder pre-trained on large-scale audio data to obtain the audio feature representation 314. The dimension of the audio feature representation 314 is (N, 2048), where N is the total number of video frames in the target video 102.

[0063] Next, a first text feature representation 315 and a second text feature representation 316 can be extracted from the caption information 306 and the description information 202, respectively, using a text encoder 312. In some embodiments, a text encoder using a CLIP model pre-trained in the domain of the target video 102 (e.g., a BERT text encoder) can be used to extract the text feature representation. The text features of the caption information 306 are taken as classification token features, with a dimension of 768. For video frames without caption information 306, they can be initialized with features all equal to 0, ultimately resulting in a first text feature representation 315 with a dimension of (N, 768). The second text feature representation 316 also has a dimension of (N, 768).

[0064] It should be noted that the dimensions of the visual feature representation 313, the audio feature representation 314, the first text feature representation 315, and the second text feature representation 316 described above are merely examples. The dimensions of these feature representations can also have other values, and this disclosure does not impose any restrictions on them.

[0065] Based on the extracted feature representations, the multimodal transformer model 318 can output a probability distribution 319 for different video segments in the target video 102. The post-processor 320, based on the probability distribution 319, can determine at least one candidate video segment 208 that matches the description information 202. In some examples, the at least one candidate video segment 208 may be in the form of a triple, including a start time point, an end time point, and a segment type.

[0066] In some embodiments, after extracting the aforementioned features, the features associated with the target video 102, namely the visual feature representation 313, the audio feature representation 314, and the first text feature representation 315, can be concatenated to obtain a video feature representation with a dimension of (N, 3328). Before fusing the video feature representation and the second text feature representation 316, a linear layer can be used to map the video feature representation and the second text feature representation 316 to the same dimension (e.g., 512-dimensional) for feature alignment. Then, in the multimodal transformer model 318, a self-attention mechanism can be used to fuse the video feature representation and the second text feature representation 316 to obtain fused features. For the prediction of at least one candidate video segment 208, the multimodal transformer model 318 can initialize M 512-dimensional query features and perform a cross-attention mechanism on the query features and the fused features, thereby predicting the start and end times of at least one candidate video segment 208.

[0067] In some embodiments, the model parameters of the visual encoder 308, audio encoder 310, and text encoder 312 remain unchanged during the training of the multimodal transformer model 318. This accelerates the training of the multimodal transformer model 318 and reduces the risk of overfitting.

[0068] In some embodiments, the target video 102 may include video related to an interactive event, and the caption information includes multiple captions indicating the progress of the interactive event. At least one candidate video segment can be determined from the video related to the interactive event, and each of the at least one candidate video segment includes one of the multiple captions. In some examples, the video related to the interactive event may include live video, game video, etc. If the target video 102 is a live video, the caption information may include information about a user entering the live stream, information about a user sending gifts to the streamer, user comments, etc. If the target video 102 is a game video, the caption information may include game status prompts, information about defeating an opponent, etc.

[0069] Figure 4 A schematic diagram of an example process 400 for determining a target video segment from candidate video segments according to some embodiments of the present disclosure is shown. Figure 4 In the example, it is assumed that the video associated with the interactive event is a game video. For example... Figure 4As shown, in box 404, video frame data 405, subtitle playback information 406, and a bounding box 407 containing the subtitle playback information 406 are extracted from candidate video segment 402. In box 408, based on the video frame data 405, subtitle playback information 406, and bounding box 407, candidate user identifier 409 and interactive user identifier information 410 can be extracted. In some examples, the identifier information located at a predetermined position (e.g., on the left side) of the bounding box 407 can be determined as candidate user identifier 409. In some examples, the interactive user identifier information 410 can be obtained at a specific position in the video frame data 405.

[0070] In box 412, the border color of candidate user identifier 409 can be detected (as an example of visual style). In some game settings, the border color of a user identifier can be used to distinguish whether the user identifier belongs to an allied user or an opposing user. For example, a blue border can be used to identify an allied user, and a red border to identify an opposing user. In box 413, if the border color is not determined to be blue, it means that candidate video clip 402 is an opposing user's highlight clip, and process 400 proceeds to box 414. In box 414, candidate video clip 402 can be deleted. In box 413, if the border color is determined to be blue, it means that candidate video clip 402 is an allied user's highlight clip, and process 400 proceeds to box 416. In box 416, it is necessary to further detect the similarity between the candidate user identifier and the identifier information of the interacting user. If the similarity is greater than a predetermined threshold, it means that candidate video clip 402 is a teammate's highlight clip, and in box 414, candidate video clip 402 can be deleted. If the similarity is not greater than a predetermined threshold, it indicates that candidate video segment 402 is its own highlight segment, and within box 418, candidate video segment 402 can be identified as the target video segment. It should be noted that determining whether a candidate user identifier matches the target user's identifier information by judging whether the user identifier's border is blue is only one example. Other colors or visual styles can also be used to determine whether a candidate user identifier matches the target user's identifier information; this disclosure does not impose any limitations on this.

[0071] Figure 5 A schematic diagram of an example process 500 for determining the identification information of a target user according to some embodiments of the present disclosure is shown. Figure 5 In the example, assume that video 501 related to the interactive event is a game video. For example... Figure 5 As shown in box 502, it can be determined from the video 501 related to the interactive event whether there is a segment showing a target user's character being defeated by an opponent, based on frame data, subtitles, or bounding box data. The segment showing a user-controlled character being defeated by an opponent can indicate that the video segment includes the target user's identification information.

[0072] If no segment is defeated by the opponent, return an empty dataset in box 504. If a segment is defeated by the opponent, extract video frame data 507, subtitle information 508, and a bounding box 509 containing the subtitle information 508 from that segment in box 506. In box 511, if other segments are defeated by the opponent, the operations in box 506 can be repeated.

[0073] In box 510, based on video frame data 507, subtitle playback information 508, and bounding box 509, the target user's identification information 512 can be extracted. For example, the identification information in frame data 507 containing a marker indicating defeat (e.g., a backslash) can be identified as the target user's identification information 512. The target user's identification information 512 can then be stored in the identification information pool 513. In box 514, low-quality identification information is filtered from the identification information pool 513. In box 516, identification information clustering is performed in the identification information pool 513. In box 518, the target user's identification information 512 can be determined.

[0074] Figure 6 A schematic diagram of an example process 600 for determining a second set of video segments according to some embodiments of the present disclosure is shown. Figure 6 In the example, assume that video 501 related to the interactive event is a game video. For example... Figure 6 As shown, in box 602, a reference video segment is extracted based on the subtitle information in video 501 related to the interactive event. This reference video segment has subtitle information that matches the description information 202. In box 604, it is determined whether the reference video segment is an existing video segment in the first group of video segments. If yes, in box 605, the next reference video segment is extracted. If not, in box 606, video frame data 607, subtitle information 608, and a bounding box 609 containing the subtitle information 608 are extracted from the reference video segment.

[0075] In box 610, based on the aforementioned video frame data 607, subtitle playback information 608, and bounding box 609, candidate user identifier 612 can be extracted. In one example, the identifier information to the left of bounding box 609 can be identified as candidate user identifier 612. In some embodiments, the method described above can be used, i.e., determining the area including subtitle playback information 608, and identifying the identifier information at a predetermined position in that area as candidate user identifier 612.

[0076] In box 614, the similarity between the candidate user identifier 612 and the target user's identifier information 512 can be calculated. In box 615, it is determined whether the candidate user identifier 612 and the target user's identifier information 512 match. If they match, in box 616, the reference video segment is assigned to the second group of video segments. If they do not match, in box 605, the next reference video segment is extracted, and based on the above process, it can be determined that the reference video frame can be assigned to the second group of video segments.

[0077] Figure 7 A flowchart of a method 700 for determining video segments according to some embodiments of the present disclosure is shown. Method 700 is implemented in... Figure 1 110 electronic devices. (Refer to...) Figure 1 The environment 100 is used to describe method 700.

[0078] In box 710, electronic device 110 acquires a target video and descriptive information associated with a target user. The target video includes video frame information, audio information, and subtitle information, and the descriptive information indicates requirements related to the extraction of segments from the target video.

[0079] In frame 720, electronic device 110 extracts visual feature representation from video frame information, audio feature representation from audio information, a first text feature representation from subtitle information, and a second text feature representation from description information.

[0080] In box 730, electronic device 110 determines at least one candidate video segment from the target video that matches the descriptive information based on visual feature representation, audio feature representation, first text feature representation, and second text feature representation.

[0081] In box 740, at least one target video segment related to the target user is determined from at least one candidate video segment, based on at least one caption information corresponding to each of the at least one candidate video segment.

[0082] In some embodiments, the target video includes a video related to the interactive event, the caption information includes multiple captions for indicating the progress of the interactive event, and determining at least one candidate video segment from the target video includes: determining at least one candidate video segment from the video related to the interactive event, each of the at least one candidate video segment including one of the multiple captions.

[0083] In some embodiments, determining at least one target video segment includes: for each of the at least one candidate video segments, determining a candidate user identifier associated with the subtitle broadcast information included in the candidate video segment; determining whether the candidate user identifier matches the identifier information of a target user; and determining the candidate video segment as the target video segment in response to the candidate user identifier matching the identifier information of the target user.

[0084] In some embodiments, determining a candidate user identifier includes: determining a region containing subtitle broadcast information from a candidate video segment; and determining the identifier information located at a predetermined position in the region as the candidate user identifier.

[0085] In some embodiments, determining whether a candidate user identifier matches the identifier information of a target user includes: determining whether a candidate user identifier matches the identifier information of a target user based on the visual style of the candidate user identifier and the similarity between the candidate user identifier and the identifier information of the interactive user corresponding to the target user.

[0086] In some embodiments, the identification information of the target user is determined by: determining a first reference video segment including the identification information of the target user from a video related to an interactive event, the first reference video segment having predetermined subtitle broadcast information or the first reference video segment being associated with an operation performed by the target user in the interactive event; and extracting the identification information of the target user from the first reference video segment.

[0087] In some embodiments, determining at least one target video segment includes: determining a first group of video segments based on subtitle information corresponding to each of at least one candidate video segment; determining a second group of video segments containing the target user's identification information from the target video based on the target user's identification information; and determining at least one target video segment by merging the first group of video segments and the second group of video segments.

[0088] In some embodiments, determining the second set of video segments includes: determining at least one second reference video segment from the target video, each second reference video segment having caption information that matches the description information; and for each of the at least one second reference video segment: determining whether a candidate user identifier associated with the caption information matches the identifier information of the target user; and in response to the candidate identifier information matching the identifier information of the target user, determining the second reference video segment as a video segment in the second set of target video segments.

[0089] In some embodiments, determining at least one target video segment includes: filtering video segments in a second set of target video segments that are duplicates of those in a first set of target video segments; and merging the filtered second target video segments with the first target video segments to determine at least one target video segment.

[0090] In some embodiments, visual feature representations are extracted using a visual encoder, audio feature representations are extracted using an audio encoder, first text feature representations and second text feature representations are extracted using a text encoder, at least one candidate video segment is determined using a multimodal transformer model, and wherein the model parameters of the visual encoder, audio encoder and text encoder remain unchanged during the training of the multimodal transformer model.

[0091] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 8 An apparatus for determining video segments according to some embodiments of the present disclosure is shown. The apparatus 800 may be implemented as or included in electronic device 110. Various modules / components in the apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0092] like Figure 8 As shown, the device 800 includes an information acquisition module 810 configured to acquire a target video and descriptive information associated with a target user. The target video includes video frame information, audio information, and subtitle information. The descriptive information indicates requirements related to segment extraction from the target video. A feature extraction module 820 is configured to extract visual feature representations from the video frame information, audio feature representations from the audio information, a first text feature representation from the subtitle information, and a second text feature representation from the descriptive information. A candidate video segment determination module 830 is configured to determine at least one candidate video segment matching the descriptive information from the target video based on the visual feature representation, audio feature representation, first text feature representation, and second text feature representation. A target video segment determination module 840 is configured to determine at least one target video segment associated with the target user from at least one candidate video segment, based at least on the subtitle information corresponding to each of the at least one candidate video segment.

[0093] In some embodiments, the target video includes video related to the interactive event, and the caption information includes multiple captions used to indicate the progress of the interactive event. The candidate video segment determination module 830 is further configured to determine at least one candidate video segment from the video related to the interactive event, each of the at least one candidate video segment including one caption from the multiple captions.

[0094] In some embodiments, the target video segment determination module 840 is further configured to, for each of the at least one candidate video segments, determine a candidate user identifier associated with the subtitle broadcast information included in the candidate video segment; determine whether the candidate user identifier matches the identifier information of the target user; and, in response to the candidate user identifier matching the identifier information of the target user, determine the candidate video segment as the target video segment.

[0095] In some embodiments, the target video segment determination module 840 is further configured to determine a region containing subtitle broadcast information from candidate video segments; and to determine the identification information located at a predetermined position in the region as a candidate user identifier.

[0096] In some embodiments, the target video segment determination module 840 is further configured to determine whether the candidate user identifier matches the identifier information of the target user based on the visual style of the candidate user identifier and the similarity between the candidate user identifier and the identifier information of the interactive user corresponding to the target user.

[0097] In some embodiments, the apparatus 800 further includes an identification information determination module configured to determine a first reference video segment including identification information of a target user from a video related to an interactive event, the first reference video segment having predetermined subtitle broadcast information or the first reference video segment being associated with an operation performed by the target user in the interactive event; and to extract the identification information of the target user from the first reference video segment.

[0098] In some embodiments, the target video segment determination module 840 is further configured to determine a first group of video segments based on subtitle information corresponding to at least one candidate video segment; determine a second group of video segments containing the target user's identification information from the target video based on the target user's identification information; and determine at least one target video segment by merging the first group of video segments and the second group of video segments.

[0099] In some embodiments, the target video segment determination module 840 is further configured to determine at least one second reference video segment from the target video, each second reference video segment having subtitle information that matches the description information; and for each of the at least one second reference video segments: determine whether a candidate user identifier associated with the subtitle information matches the identifier information of the target user; and in response to the candidate identifier information matching the identifier information of the target user, determine the second reference video segment as a video segment in the second set of target video segments.

[0100] In some embodiments, the target video segment determination module 840 is further configured to filter video segments in the second group of target video segments that are duplicates of those in the first group of target video segments; and to merge the filtered second target video segments with the first target video segments to determine at least one target video segment.

[0101] In some embodiments, visual feature representations are extracted using a visual encoder, audio feature representations are extracted using an audio encoder, first text feature representations and second text feature representations are extracted using a text encoder, at least one candidate video segment is determined using a multimodal transformer model, and wherein the model parameters of the visual encoder, audio encoder and text encoder remain unchanged during the training of the multimodal transformer model.

[0102] The units and / or modules included in device 800 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 800 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0103] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may include, for example, […]. Figure 1 Electronic device 110.

[0104] Figure 9 A block diagram of an electronic device 900 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 9 The electronic device 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 9 The electronic device 900 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 8 Device 800.

[0105] like Figure 9 As shown, electronic device 900 is in the form of a general-purpose electronic device. Components of electronic device 900 may include, but are not limited to, one or more processing units or processors 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 900.

[0106] Electronic device 900 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 900.

[0107] Electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0108] The communication unit 940 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 900 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0109] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 900 can also communicate with one or more external devices (not shown) via communication unit 940 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 900, or with any device that enables electronic device 900 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0110] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0111] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0112] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0113] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0115] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for determining a video segment, comprising: Obtain a target video and descriptive information associated with a target user, wherein the target video includes video frame information, audio information, and subtitle information, and the descriptive information indicates requirements related to segment extraction from the target video; Visual feature representations are extracted from the video frame information, audio feature representations are extracted from the audio information, a first text feature representation is extracted from the subtitle information, and a second text feature representation is extracted from the description information; Based on the visual feature representation, the audio feature representation, the first text feature representation, and the second text feature representation, at least one candidate video segment matching the description information is determined from the target video; and At least one target video segment related to the target user is determined from the at least one candidate video segment, based on the subtitle information corresponding to each of the at least one candidate video segment.

2. The method of claim 1, wherein the target video includes a video related to the interactive event, the caption information includes multiple captions for indicating the progress of the interactive event, and determining the at least one candidate video segment from the target video includes: At least one candidate video segment is determined from the video related to the interactive event, and each of the at least one candidate video segment includes one of the multiple caption broadcast messages.

3. The method of claim 2, wherein determining the at least one target video segment comprises: For each of the at least one candidate video segments, Determine the candidate user identifier associated with the subtitle broadcast information included in the candidate video segment; Determine whether the candidate user identifier matches the identifier information of the target user; In response to the candidate user identifier matching the identifier information of the target user, the candidate video segment is determined as the target video segment.

4. The method of claim 3, wherein determining the candidate user identifier comprises: Determine the region containing the subtitle broadcast information from the candidate video segments; as well as The identification information located at a predetermined location in the area is determined as the candidate user identifier.

5. The method according to claim 3, wherein determining whether the candidate user identifier matches the identifier information of the target user includes: Based on the visual style of the candidate user identifier and the similarity between the candidate user identifier and the identifier information of the interactive user corresponding to the target user, it is determined whether the candidate user identifier matches the identifier information of the target user.

6. The method of claim 3, wherein the identification information of the target user is determined by: Determine a first reference video segment from the video related to the interactive event, including the identification information of the target user, wherein the first reference video segment has predetermined subtitle information or is associated with an operation performed by the target user in the interactive event; and Extract the identification information of the target user from the first reference video segment.

7. The method of claim 1, wherein determining the at least one target video segment comprises: Based on the subtitle information corresponding to each of the at least one candidate video segment, a first group of video segments is determined; Based on the target user's identification information, a second set of video segments containing the target user's identification information is determined from the target video; The at least one target video segment is determined by merging the first group of video segments and the second group of video segments.

8. The method of claim 7, wherein determining the second group of video segments comprises: At least one second reference video segment is determined from the target video, and each second reference video segment has subtitle information that matches the description information; as well as For each of the at least one second reference video segment: Determine whether the candidate user identifier associated with the subtitle information matches the identifier information of the target user; and In response to the candidate user identifier matching the identifier information of the target user, the second reference video segment is determined as a video segment in the second group of target video segments.

9. The method of claim 7, wherein determining the at least one target video segment comprises: Filter out video segments in the second group of target video segments that are duplicates of those in the first group of target video segments; as well as The filtered second group of target video segments is merged with the first group of target video segments to determine the at least one target video segment.

10. The method of claim 1, wherein the visual feature representation is extracted using a visual encoder, the audio feature representation is extracted using an audio encoder, the first text feature representation and the second text feature representation are extracted using a text encoder, and the at least one candidate video segment is determined using a multimodal transformer model, and During the training of the multimodal transformer model, the model parameters of the visual encoder, the audio encoder, and the text encoder remain unchanged.

11. An apparatus for determining a video segment, comprising: The information acquisition module is configured to acquire a target video and descriptive information associated with a target user. The target video includes video frame information, audio information, and subtitle information. The descriptive information indicates requirements related to the extraction of segments from the target video. The feature extraction module is configured to extract visual feature representations from the video frame information, extract audio feature representations from the audio information, extract a first text feature representation from the subtitle information, and extract a second text feature representation from the description information; A candidate video segment determination module is configured to determine at least one candidate video segment from the target video that matches the description information, based on the visual feature representation, the audio feature representation, the first text feature representation, and the second text feature representation; and The target video segment determination module is configured to determine at least one target video segment related to the target user from the at least one candidate video segment, based at least on the subtitle information corresponding to each of the at least one candidate video segment.

12. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the device to perform the method according to any one of claims 1 to 10 when executed by the at least one processor.

13. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 10.

14. A computer program product comprising computer-executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images